From 99a25967e50f5c0e9f28d8249fa58d4f2542330d Mon Sep 17 00:00:00 2001 From: Bolin Chen Date: Tue, 28 Jul 2026 21:29:17 -0700 Subject: [PATCH] feat(computer-use): native desktop automation for macOS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Read and drive native desktop apps through the macOS accessibility layer: list on-screen windows, snapshot one as a numbered element tree, then click, type, set values, scroll, drag or run a named action by element index or screen coordinate. Pure ctypes over AXUIElement / CoreGraphics / ImageIO — no pyobjc. Windows and Linux report unsupported rather than degrading. Computer use is ONE operator opt-in. The enable lives on the keystone `computer_use.json`, which `security._SENSITIVE_HOME_DIRS` fences the agent away from, so a prompt-injected agent can neither read nor flip it. Past that point the agent drives the desktop the way the operator would: there is no governance model, no per-app allow-list ceiling, no unattended-surface refusal, and no interactive-approval floor. That is a product decision, and the residual risk is documented rather than implied — see `docs/system-specs/modules/computer-use.md` -> "What is enforced, and what is not" and "The keystone is the whole security boundary". What still refuses, all enforced in band on the dispatch path: - KiroCrew's own window, because driving our Settings UI would route around the keystone that holds the enable; - password fields — never read, and a window holding one is never captured; - sensitive-text and secure-target checks on every input verb; - credential redaction on the way out. The real-pointer path (`click_method: "global"`) needs no second opt-in, but the model must NAME it: `auto` never resolves onto a pointer-moving method, so the operator's cursor is never warped by accident, and every such gesture gets its own SEL `tool_kind` so "did the agent take my mouse?" is one log filter. With the ceiling gone the audit trail is the accountability: every call is recorded, allowed or refused. One accessibility walk now reads what a model needs to act on the first try, rather than leaving it to guess a coordinate and read back a screenshot: - element frames (`AXPosition`/`AXSize`, unboxed from `AXValue` with the type checked so a CGPoint can never be transposed into a CGSize), reported WINDOW-LOCAL with the window origin published alongside them — the screenshot is a crop of the window, so a screen-absolute rect could not be related to any pixel the model can see; - `editable` / `selected` / `expanded` traits, read as a tri-state so absent never renders as false. `editable` comes from AXUIElementIsAttributeSettable, not AXEnabled: a read-only field reports enabled with a readable value, so the model used to type into it, get an ok result, and lose the text; - the focused element and the user's text selection, read once per walk off the application element (never system-wide, which would follow the operator instead of the target app) and compared with CFEqual, since AXFocusedUIElement returns a fresh reference that no address comparison would ever match; - `AXRows` / `AXVisibleChildren` merged with `AXChildren`, deduplicated by element identity. A table, outline or list often exposes its rows ONLY there, so a children-only walk rendered a spreadsheet, a Finder list or a mail inbox as an empty container. A secure element still discloses only its existence — no traits and no frame, for the same reason its value is withheld. The click ladder gains a last rung: when the addressed element refuses every verb, press the enclosing control. Web content renders a clickable row as a plain AXStaticText inside a pressable wrapper, which left the whole row dead to an element click. Bounded by hops AND by area ratio, declining outright when the target has no frame to compare against, never for a right click, and the result says it pressed the container so the model can tell "my click worked" from "something near my click worked". Also fixes a latent bug found while wiring this up: capture_snapshot_image rebuilt the frozen Snapshot field by field, so every field added afterwards was dropped whenever a screenshot was attached — and only then, which is why it had gone unnoticed. It uses dataclasses.replace now, pinned by a test that walks the whole dataclass so the next field is covered without anyone remembering. The Windows shard's two failures were the new data-home pin test asserting POSIX-only semantics: Path("/usr").resolve() is :\usr there, which the resolver rightly accepts. Split into a portable root case and a POSIX-gated system-directory case. GPT 5.6 found two real blockers on the previous SHA; both are fixed rather than overridden. `sky_click` silently downgraded right and middle clicks to left. The recipe took no button argument at all and built the left-button event codes unconditionally, and nothing upstream refused the pair — so a right-click request through this method activated the control instead of opening its context menu, on a background window the operator cannot see. Refused now, not implemented: the private sequence was reverse-engineered for a left click and the button number is one field among nine, so a right-click variant would be invented rather than observed. Gated at the dispatch chokepoint (policy.check_method_button, on the RESOLVED method) and re-checked inside macos_skylight; the driver now passes the button rather than assuming it, pinned by a test that reads the call site because a behavioural test alone would keep passing if the argument were dropped again. A malformed keystone turned the Settings GET into an HTTP 500. load_policy_config raises on a present-but-malformed allowed_apps by design — coercing it to empty would convert an operator's restriction into no restriction — but on the READ path that escaped _snapshot() and made the only UI that can repair the file unreachable. The page has to render precisely because the file is broken. It now falls back to an empty PolicyConfig and publishes policy_error, which the panel renders as a warning naming the file, since an empty allow-list otherwise reads as "no restriction configured". The ceiling is unchanged: every dispatch still loads the policy itself and still refuses, and a test asserts both halves. Also rewrites the CHANGELOG entries, which still advertised the governance model, per-app narrowing and per-use approval that the scope change deleted — GPT's third finding, and correct. A third GPT blocker, also real: the mutating tools' action header bypassed credential redaction. Every mutator returns "\n\nRefreshed state:\n"; the tree half is redacted inside render_tree, but the header was concatenated after that pass — and detail is not our prose. Every driver confirmation interpolates app-supplied text (_click_text embeds app.name, the process name macOS reports), so a process named "Notes key=AKIA…" put a raw credential directly in front of a fully redacted tree. detail is now redacted at the interpolation. Deliberately NOT by redacting the joined string: render_tree appends its screenshot note after its own pass because the per-user temp path contains a long random segment the bare-secret-key heuristic masks, so a second pass would replace every screenshot path with a placeholder and that channel would silently stop working. Header redacted alone, already-redacted body untouched — both halves pinned by tests, the second one structurally, since a mutator's refresh walk carries no image and so no behavioural test in that file would catch the regression. Also fixes a stale skill contract GPT flagged as advisory: SKILL.md advertised computer_type_text(app, text, element_index?) with an "else the focused control" fallback, which the runtime has never allowed. element_index is REQUIRED on both keyboard tools because an unnamed target has no role or subrole for the secure-field check to inspect, and press_key("tab") can move focus onto a password box. GPT prescribed changing the schema to match the doc; taken the other way round, since the runtime behaviour is the security control. A test now pins the doc against the runtime so the two cannot drift again. The same stale claim lived in the spec too, in the more dangerous direction: "What no longer refuses" listed indexless keyboard input as working again, which was written during the scope change and never implemented — so a reader auditing the security posture from that document would conclude a control was gone that is still enforced. Moved to "What still refuses" alongside two rows that were also missing (the action header's redaction, and the non-left sky_click refusal), and pinned by a test so neither document can drift from the runtime again. A fourth GPT blocker, also real: the bundle Info.plist read was check-then-open. It called is_sensitive_path and then opened the path in a separate step, so a final-component symlink swapped in between would read a protected file's bytes on a path that never touched the hardened gate — and the agent chooses which process to target, so it can arrange the bundle. It now reads through hooks.safe_read_prefix, which canonicalizes with realpath, re-checks the RESOLVED target and opens with O_NOFOLLOW; that helper is the repo's stated requirement for any read of an agent-influenced path. The size cap moved off a getsize stat and onto the bytes actually read, since statting a path and then opening it is the same raceable shape. Reading MAX_INFO_PLIST_BYTES + 1 is what distinguishes "at the limit" from "over it" without a second stat. The existing floor test patched kiro_crew.security.is_sensitive_path, which the helper resolves through its own import — so it would have passed against a bypassed floor. Rewritten to stage a plist genuinely under ~/.ssh with $HOME redirected, plus a symlinked-plist case, an oversized case, and a structural assertion that no bare open() returns here (the original bug passed every behavioural test in that file). Fixes the Windows shard failure my own plist test introduced: it redirected only $HOME, but os.path.expanduser reads USERPROFILE there, so the planted bundle was not under a sensitive dir and the test asserted the opposite of what it meant. Both vars are set now, and the symlink case is POSIX-gated (creating a symlink needs elevation on Windows; the resolved-target check it exercises is platform-independent and covered by the sensitive-dir case). A fifth GPT blocker was half right, and the half that was right is fixed without taking its prescription. An unresolved session key was forwarded as the empty string, and SnapshotIndex namespaces by (session_key, window_key) — so every unresolved session shared one ("", window) slot. Unresolved is the NORMAL case on macOS, so two concurrent sessions observing the same window overwrote each other's element indices, and each one's own verify_fingerprint still passed because both trees describe the same window: a wrong-target action with nothing reporting it. GPT prescribed refusing an empty key. Not taken — that is the refusal removed by product decision, and it is what made the feature unusable on its only supported platform. Fixed by namespacing instead: an unresolved key becomes unresolved:, and kiro-cli spawns one shim per session, so the pid separates the namespaces exactly as far as the sessions really are separate. Read at call time so a forked child cannot inherit its parent's string and re-alias. The prefix keeps it legible as a namespace separator rather than attribution. Nothing is refused; only the cache key changed. --- AGENTS.md | 41 +- CHANGELOG.md | 3 + NOTICE | 8 + config-baseline.json | 120 + docs/system-specs/index.md | 1 + docs/system-specs/modules/cli.md | 64 + docs/system-specs/modules/computer-use.md | 1514 ++++++++++++ docs/system-specs/modules/config.md | 75 + docs/system-specs/modules/governance.md | 146 +- docs/system-specs/modules/security.md | 88 + scripts/scrub-allowlist.txt | 7 + src/kiro_crew/agent.py | 102 + .../builtin_skills/computer-use/SKILL.md | 371 +++ .../prepare-pr/scripts/diff_signals.py | 21 +- .../prepare-pr/scripts/enable_automerge.py | 28 +- .../prepare-pr/scripts/preflight.py | 27 +- .../prepare-pr/scripts/resolve_profile.py | 1 + src/kiro_crew/cli.py | 41 + src/kiro_crew/cli_doctor.py | 26 +- src/kiro_crew/computer_use/__init__.py | 72 + src/kiro_crew/computer_use/apps_macos.py | 486 ++++ src/kiro_crew/computer_use/backend.py | 396 ++++ src/kiro_crew/computer_use/capture_macos.py | 294 +++ src/kiro_crew/computer_use/cli.py | 485 ++++ src/kiro_crew/computer_use/cursor_motion.py | 425 ++++ src/kiro_crew/computer_use/enable_state.py | 131 ++ src/kiro_crew/computer_use/gate.py | 259 +++ src/kiro_crew/computer_use/index.py | 296 +++ src/kiro_crew/computer_use/keymap.py | 188 ++ src/kiro_crew/computer_use/linux_driver.py | 49 + src/kiro_crew/computer_use/macos_driver.py | 821 +++++++ src/kiro_crew/computer_use/macos_ffi.py | 2040 ++++++++++++++++ src/kiro_crew/computer_use/macos_skylight.py | 669 ++++++ src/kiro_crew/computer_use/overlay.py | 537 +++++ src/kiro_crew/computer_use/overlay_proc.py | 768 ++++++ src/kiro_crew/computer_use/permissions.py | 195 ++ src/kiro_crew/computer_use/policy.py | 437 ++++ src/kiro_crew/computer_use/render.py | 304 +++ src/kiro_crew/computer_use/screencast.py | 333 +++ src/kiro_crew/computer_use/service.py | 536 +++++ src/kiro_crew/computer_use/snapshot_macos.py | 1022 ++++++++ src/kiro_crew/computer_use/tools.py | 1147 +++++++++ src/kiro_crew/computer_use/types.py | 1060 +++++++++ src/kiro_crew/computer_use/windows_driver.py | 48 + src/kiro_crew/config/defaults.json | 3 +- src/kiro_crew/config/loader.py | 193 ++ src/kiro_crew/config/prompt.md | 27 + src/kiro_crew/dashboard/handlers/__init__.py | 10 + .../dashboard/handlers/computer_use.py | 964 ++++++++ src/kiro_crew/dashboard/handlers/core.py | 22 + src/kiro_crew/dashboard/handlers/mcp.py | 7 +- src/kiro_crew/dashboard/server.py | 35 + src/kiro_crew/hooks.py | 55 + src/kiro_crew/mcp_cleanup.py | 2 +- src/kiro_crew/mcp_computer.py | 674 ++++++ src/kiro_crew/mcp_discovery.py | 6 +- src/kiro_crew/onboarding_import.py | 3 + src/kiro_crew/platform/governance.py | 119 + src/kiro_crew/security.py | 26 +- src/kiro_crew/security_posture.py | 47 +- src/kiro_crew/testing/fake_computer_use.py | 633 +++++ src/kiro_crew/validation.py | 202 ++ .../computer-use/settings-default-off.png | Bin 0 -> 398958 bytes .../computer-use/settings-enabled.png | Bin 0 -> 452464 bytes test/conftest.py | 39 + test/test_cli.py | 89 +- test/test_computer_use_api.py | 1488 ++++++++++++ test/test_computer_use_apps.py | 738 ++++++ test/test_computer_use_backend.py | 437 ++++ test/test_computer_use_capture.py | 771 ++++++ test/test_computer_use_cli.py | 380 +++ test/test_computer_use_cursor_motion.py | 478 ++++ test/test_computer_use_enable_state.py | 366 +++ test/test_computer_use_ffi.py | 1925 +++++++++++++++ test/test_computer_use_ffi_argtypes.py | 468 ++++ test/test_computer_use_gate.py | 223 ++ test/test_computer_use_overlay.py | 1270 ++++++++++ test/test_computer_use_registration.py | 691 ++++++ test/test_computer_use_skylight.py | 354 +++ test/test_computer_use_snapshot.py | 794 +++++++ test/test_computer_use_snapshot_macos.py | 2065 +++++++++++++++++ test/test_computer_use_unsupported.py | 330 +++ test/test_config_loader.py | 24 +- test/test_identity_topology.py | 7 + test/test_mcp_computer.py | 2028 ++++++++++++++++ test/test_spawn_audit.py | 30 + website/scripts/settingsExtract.ts | 1 + website/src/App.tsx | 8 + website/src/api/client.ts | 66 + .../src/components/ComputerUseLiveView.tsx | 422 ++++ .../providers/settingsRegistry.test.ts | 2 +- .../commandPalette/settingsKeywords.ts | 4 + .../commandPalette/settingsRegistry.gen.ts | 37 + website/src/hooks/useComputerUseFrame.ts | 84 + website/src/hooks/useWebSocket.ts | 10 + website/src/i18n/locales/en.manual.json | 4 + website/src/i18n/locales/zh-CN.json | 4 + website/src/pages/SettingsPage.tsx | 5 +- .../pages/settings/ComputerUsePanel.test.tsx | 234 ++ .../src/pages/settings/ComputerUsePanel.tsx | 332 +++ website/src/test/ComputerUseLiveView.test.tsx | 241 ++ website/src/test/SettingsPage.test.tsx | 14 + 102 files changed, 35087 insertions(+), 86 deletions(-) create mode 100644 docs/system-specs/modules/computer-use.md create mode 100644 src/kiro_crew/builtin_skills/computer-use/SKILL.md create mode 100644 src/kiro_crew/computer_use/__init__.py create mode 100644 src/kiro_crew/computer_use/apps_macos.py create mode 100644 src/kiro_crew/computer_use/backend.py create mode 100644 src/kiro_crew/computer_use/capture_macos.py create mode 100644 src/kiro_crew/computer_use/cli.py create mode 100644 src/kiro_crew/computer_use/cursor_motion.py create mode 100644 src/kiro_crew/computer_use/enable_state.py create mode 100644 src/kiro_crew/computer_use/gate.py create mode 100644 src/kiro_crew/computer_use/index.py create mode 100644 src/kiro_crew/computer_use/keymap.py create mode 100644 src/kiro_crew/computer_use/linux_driver.py create mode 100644 src/kiro_crew/computer_use/macos_driver.py create mode 100644 src/kiro_crew/computer_use/macos_ffi.py create mode 100644 src/kiro_crew/computer_use/macos_skylight.py create mode 100644 src/kiro_crew/computer_use/overlay.py create mode 100644 src/kiro_crew/computer_use/overlay_proc.py create mode 100644 src/kiro_crew/computer_use/permissions.py create mode 100644 src/kiro_crew/computer_use/policy.py create mode 100644 src/kiro_crew/computer_use/render.py create mode 100644 src/kiro_crew/computer_use/screencast.py create mode 100644 src/kiro_crew/computer_use/service.py create mode 100644 src/kiro_crew/computer_use/snapshot_macos.py create mode 100644 src/kiro_crew/computer_use/tools.py create mode 100644 src/kiro_crew/computer_use/types.py create mode 100644 src/kiro_crew/computer_use/windows_driver.py create mode 100644 src/kiro_crew/dashboard/handlers/computer_use.py create mode 100644 src/kiro_crew/mcp_computer.py create mode 100644 src/kiro_crew/testing/fake_computer_use.py create mode 100644 temp-screenshots/computer-use/settings-default-off.png create mode 100644 temp-screenshots/computer-use/settings-enabled.png create mode 100644 test/test_computer_use_api.py create mode 100644 test/test_computer_use_apps.py create mode 100644 test/test_computer_use_backend.py create mode 100644 test/test_computer_use_capture.py create mode 100644 test/test_computer_use_cli.py create mode 100644 test/test_computer_use_cursor_motion.py create mode 100644 test/test_computer_use_enable_state.py create mode 100644 test/test_computer_use_ffi.py create mode 100644 test/test_computer_use_ffi_argtypes.py create mode 100644 test/test_computer_use_gate.py create mode 100644 test/test_computer_use_overlay.py create mode 100644 test/test_computer_use_registration.py create mode 100644 test/test_computer_use_skylight.py create mode 100644 test/test_computer_use_snapshot.py create mode 100644 test/test_computer_use_snapshot_macos.py create mode 100644 test/test_computer_use_unsupported.py create mode 100644 test/test_mcp_computer.py create mode 100644 website/src/components/ComputerUseLiveView.tsx create mode 100644 website/src/hooks/useComputerUseFrame.ts create mode 100644 website/src/pages/settings/ComputerUsePanel.test.tsx create mode 100644 website/src/pages/settings/ComputerUsePanel.tsx create mode 100644 website/src/test/ComputerUseLiveView.test.tsx diff --git a/AGENTS.md b/AGENTS.md index b9a1aad8043..5f0c79e90d1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -90,6 +90,31 @@ infrastructure that survives the upstream sync. Full map: so the agent cannot read/write its own ceiling — the single mechanism that makes the ceiling un-disableable. When editing `security.py`'s sensitive-path or bash-command matchers, keep these covered (incl. write/extract verbs). +- **Computer use is deliberately NOT governed.** It is ONE operator opt-in on the + keystone `computer_use.json` (fenced by `security._SENSITIVE_HOME_DIRS`, so the + agent can neither read nor flip it). There are **no** `computer_use.*` scopes, no + `capabilities.computer_use*` rows, no approval ordinal and no pointer permit — an + earlier revision had all eight and they were removed by product decision. Do NOT + reintroduce them without that decision being reversed; `gate.py` is audit-only and + always permits. + What still refuses, all enforced **in band** on the `tools._dispatch` path (never + at the `hooks` PreToolUse gate, which is fail-OPEN and can be skipped by a + pre-authorized tool): the keystone enable, **KiroCrew's own window** + (`policy.check_app` — driving our Settings UI would route around the keystone), + secure/password fields, the sensitive-text scan, and credential redaction. Keep + these on the dispatch path. Secure-field redaction is an **always-on floor with no + policy key** — never make it governable. + **Element-targeted, non-pointer input is the DEFAULT.** `click_method: "global"` is + the only path that warps the operator's REAL pointer; it needs no separate opt-in, + but the model MUST NAME it — `auto` must NEVER resolve onto it (load-bearing + invariant with its own test; it is now the only thing between an ordinary click and + the user's cursor). Every such gesture is SEL-audited under its own `tool_kind`. + `sky_click` is deliberately not ported (private SkyLight API). + Flipping the enable RESTARTS chat sessions (`_reset_all_sessions`), because + kiro-cli caches `tools/list` per session and ACP has no `tools/list_changed`. + Keep prose in sync: `config/prompt.md` and `builtin_skills/computer-use/SKILL.md` + ship to users and must not describe refusals that no longer exist. See + `docs/system-specs/modules/computer-use.md` + `.../governance.md`. - Keep the `ACP_BACKEND_CLAUDE` seam and `platform/` extension points intact; don't add public registration glue, and keep the stubs stubbed. @@ -371,6 +396,7 @@ Jane Doe (janedoe), John Smith (jsmith) | `cli_server.py` | CLI gateway/server commands (split from cli.py) | | `cli_setup.py` | CLI setup wizard (split from cli.py) | | `dashboard/chat_runner.py` | Chat execution logic (split from `dashboard/chat.py`) | +| `computer_use/` | Native desktop GUI automation (macOS today; Windows/Linux refuse). `ComputerUseBackend` ABC + swap registry, the keystone primary enable, the audit-only `gate.py`, the in-band refusals in `policy.py`, and the ctypes driver — all in-gateway native work confined to `macos_ffi.py`. The **one documented exception**: `overlay_proc.py` (the Cursor Motion AppKit child) has its own ctypes surface because AppKit needs a main-thread run loop and the gateway's main thread is the asyncio loop, so it MUST be out of process. See `docs/system-specs/modules/computer-use.md`. | | `platform/` | **Composed Platform Providers (CPP) seam + Governance model** — see the dedicated section below. | ### Platform layer: Composed Platform Providers (CPP) + Governance @@ -496,10 +522,11 @@ KiroCrew exposes capabilities to the LLM via two mechanisms: 1. **MCP tools** (native): kiro-cli calls them directly with structured JSON params — **preferred for all LLM-facing operations** - `kirocrew-cron` MCP server: `cron_list`, `cron_add`, `cron_update`, `cron_remove`, `cron_remove_all`, `cron_pause`, `cron_resume`, `cron_trigger` - `kirocrew-core` MCP server: `spawn_run`, `spawn_list`, `spawn_status`, `learn_add`, `learn_list`, `learn_remove`, `task_run`, `wait`, `register_hook`, `send_message`, `send_notification`, `local_knowledge_search` + - `kirocrew-computer` MCP server (10 tools): `computer_list_apps`, `computer_get_state`, `computer_click`, `computer_drag`, `computer_type_text`, `computer_press_key`, `computer_set_value`, `computer_scroll`, `computer_perform_action`, `computer_end_turn` — native desktop GUI automation. Default-OFF behind a keystone primary enable (`~/.kiro/crew/computer_use.json`, NOT `config.json`); the stdio process is a thin shim and the authoritative fail-closed gate runs in the gateway. `computer_click` takes **either** `element_index` **or** `x`+`y` (never both) plus optional `click_count` (1-3), `mouse_button` (`left`/`right`/`middle`) and `click_method` (`auto`/`accessibility`/`app_post`/`global`); `computer_drag` is coordinate-only. Both keyboard tools (`computer_type_text`, `computer_press_key`) REQUIRE `element_index`: an unnamed target has no role/subrole, so the always-on secure-field (password) refusal could not inspect it. See `docs/system-specs/modules/computer-use.md`. - `playwright` MCP server (`@playwright/mcp`): `browser_navigate`, `browser_click`, `browser_snapshot`, `browser_take_screenshot`, `browser_fill_form`, `browser_type`, `browser_press_key`, `browser_evaluate`, `browser_hover`, `browser_drag`, `browser_select_option`, `browser_tabs`, `browser_close`, `browser_wait_for`, `browser_resize` - `slack-mcp` (mcpServers): Slack integration - Configured in `agents/defaults.json` → `mcpServers` → installed to `kirocrew.json` - - `kirocrew-cron` and `kirocrew-core` are managed MCP servers in `agent.py:_MANAGED_MCP_SERVERS` — auto-registered, refreshed preserving user customizations + - `kirocrew-cron`, `kirocrew-core` and `kirocrew-computer` are managed MCP servers in `agent.py:_MANAGED_MCP_SERVERS` — auto-registered, refreshed preserving user customizations. `kirocrew-computer` is deliberately added to `tools` but **NOT** `allowedTools`, and its managed spec carries **no `autoApprove` key** (an autoApproved MCP tool never reaches `hooks.on_tool_call`) - MCP discovery (`mcp_discovery.py`): on-demand only — users trigger from dashboard "Discover & Sync" button 2. **Skills** (`skills/*/SKILL.md`): on-demand knowledge files for specialized workflows @@ -557,6 +584,17 @@ should always use the MCP tool equivalents. | — | `artifact_post_comment` | kirocrew-core | | — | `artifact_mark_review` | kirocrew-core | | — | `artifact_delete_comment` | kirocrew-core | +| `kirocrew computer apps` | `computer_list_apps` | kirocrew-computer | +| — | `computer_get_state` | kirocrew-computer | +| — | `computer_click` | kirocrew-computer | +| — | `computer_drag` | kirocrew-computer | +| — | `computer_type_text` | kirocrew-computer | +| — | `computer_press_key` | kirocrew-computer | +| — | `computer_set_value` | kirocrew-computer | +| — | `computer_scroll` | kirocrew-computer | +| — | `computer_perform_action` | kirocrew-computer | +| — | `computer_end_turn` | kirocrew-computer | +| `kirocrew computer call ` / `call --calls '[…]'` | — (deliberately none) | — | | — | `browser_navigate` | playwright | | — | `browser_click` | playwright | | — | `browser_snapshot` | playwright | @@ -568,6 +606,7 @@ should always use the MCP tool equivalents. - **Handler keywords**: only for instant user-typed commands with no LLM round-trip (e.g. `cron list`, `spawn list`) - **Do NOT** add regex to match NL variants — the LLM handles NL interpretation +- **`kirocrew computer call` is the one deliberate "no MCP twin" row.** It is not a capability — it is a human debug/repro harness that runs the ten existing `computer_*` tools through the same gated chokepoint (optionally a JSON array of them in ONE process, so `element_index` values stay resolvable). The MCP-first rule exists so the model gets a structured tool instead of shelling out, and the model already has all ten. A tool that runs other tools would let a model launder one per-call gate decision into many — so do NOT add `computer_call`. #### Project-Level Configuration diff --git a/CHANGELOG.md b/CHANGELOG.md index e150720172f..da2faf655ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,9 @@ All notable changes to KiroCrew are documented in this file. - **Agent templates can map skills** — Agent Capabilities → Agent Templates gains a **Skills** section: pick which skills an agent template loads, add and remove them inline, and see the count on the agent list. Previously there was no way to do this at all — the skill count shown next to every agent read zero because it only recognised an internal `builder-mcp --skill-name-filter` convention, `PATCH /api/agents/detail/{name}` accepted nothing but `model`, and at runtime the choice was all-or-nothing: the `kirocrew` agent received the entire catalog while every custom agent received none. A mapping is stored the kiro-cli-native way, as `skill://` entries in the agent's `resources`, so kiro-cli loads the SKILL.md files itself when the agent starts, and KiroCrew's own injected skills block narrows to the same set on the Claude Code backend (where agent `resources` are not read). Agents with no mapping keep exactly their old behavior. `file://` steering globs are never touched, and hand-authored `skill://` entries the editor cannot express — wildcards, or paths outside the known skill roots — are shown read-only and preserved across edits. See `docs/agents.md`. - **Official Docker image** — The gateway now ships as a multi-arch container at `ghcr.io/kirodotdev/kirocrew` (`stable` / `insider` / `nightly` channel tags plus immutable version tags, linux/amd64 + linux/arm64), built from the exact same wheel pip users install and carrying SLSA build provenance. Registry access remains private for now and requires GHCR authentication with package access. One `docker run` with a single volume gives an always-on headless gateway — dashboard, Slack/Discord/Telegram/WeCom/Webex bots, crons — with kiro-cli preinstalled; `docker exec` in for the one-time `kiro-cli login` and to mint dashboard links. A new `KIROCREW_BIND` env override (validated, fail-narrow) lets the gateway bind beyond loopback inside the container's network namespace so published ports actually work; token auth, CSRF, and Host validation are unchanged and every request still needs a dashboard token. Orchestrator liveness/readiness probes (`/api/health`, `/api/live`, `/api/ready`) are now reachable when addressed by container/pod IP: they bypass Host validation, and in exchange the build-identity fields are additionally gated on a served Host header — a DNS-rebound loopback request learns only `{"ok": true}`. See `docs/DOCKER.md`. +- **Computer use — let the agent drive your desktop apps** — KiroCrew can now read and operate native applications through the accessibility layer, not just the browser: list what's on screen, get one window as a structured, numbered outline of its buttons, fields, rows and menus, then act on an element by number — press it, type into it, set its value, scroll it, or run one of its own named actions. This reaches the work that lives outside a browser tab: pull a figure out of a spreadsheet into a report, walk a desktop-only internal tool, refile a batch of rows in a native app, or read an error dialog and tell you what it says. The structured outline is the primary channel, so a turn is a few thousand tokens rather than a screenshot dump; an optional compressed screenshot is written to disk and only its path is handed over, to be opened if the outline isn't enough. **Your pointer stays where you left it by default** — actions go to the target application directly, so your cursor, your focused window and your keyboard are left alone (see the entry below for the opt-in exception and for coordinate clicking and dragging). **Off by default and macOS-only in this release** (Windows and Linux say so clearly rather than half-working): turn it on in Settings → Computer Use, which also shows the macOS Accessibility and Screen Recording permission state. It is one switch, and it is the only one: once you turn it on, the agent drives your desktop the way you would. That switch lives in a file the agent can neither read nor write, so a prompt-injected agent cannot enable it — but past that point there is no per-app allow-list and no per-action approval prompt. What still refuses: **password fields are never read and a window holding one is never photographed**, text that looks like a credential is never typed, and **KiroCrew's own dashboard is refused outright** (driving our own Settings would route around the switch that holds the enable — including the dashboard open in a browser tab). Everything else, a terminal included, is reachable. That is a deliberate choice for a single-user machine where you are trusted with your own desktop, and the trade is written up in full in `docs/system-specs/modules/computer-use.md`. Every call is written to the audit log, allowed or refused — with the feature ungoverned, that record is the accountability. + +- **Computer use reaches the UI that has no buttons — coordinates, dragging, and an optional visible cursor** — Some things on a screen simply are not controls: a drawing canvas, a map, a timeline, a chart, a slider, a custom-drawn panel in an old internal tool. The agent can now click a *point* in an app (`x`, `y`) and drag between two points, so it can sweep a range, reorder a list, stroke a canvas or drag a slider — plus right-click and middle-click, and double- and triple-click for select-word and select-line. Addressing a control by its number from the outline is still the default and still what you should prefer: it is checked against the window moving underneath it, while a coordinate lands on whatever happens to be at that spot when it arrives. **Your mouse pointer still does not move.** A coordinate click is delivered straight to the target application, so it works on a background window without stealing your cursor, your focus or your keyboard. For the rare UI that only responds to a physically real click — a Dock item, a menu-bar extra — the agent can take your actual pointer — but **only if it asks for that path by name**. The automatic choice never resolves onto it, so your cursor cannot be warped as a side effect of an ordinary click, and every such gesture is logged under its own kind so "did the agent take my mouse?" is one filter in the audit log. There is also a third path for the opposite problem: clicking a window that is **behind** another one, without raising it and without touching your pointer, for renderers that ignore a delivered click because they check with the window server which window is in front. Two new ways to *watch* what is happening, both purely for you and neither granting the agent anything: a floating **live view** panel mirrors the screenshots the agent takes (never a fresh capture of its own, and never a window holding a password field), and an optional **Cursor Motion** overlay draws a moving cursor on your real desktop along the path a click is about to take, so you can see where the agent is aiming before it acts. The overlay is deliberately invisible to screenshots — including the agent's own — so it can never end up in the pixels the agent is reading. For debugging and bug reports there is also a new `kirocrew computer call` command that runs one tool, or a whole sequence in a single pass, through exactly the same permission checks the agent goes through, so a problem can be reproduced from a terminal without a model in the loop. - **WeCom channel settings panel** — Settings gains a WeCom tab so the WeCom (企业微信) channel can be set up and modified from the dashboard instead of hand-editing `.env` and `config.json`: paste the Bot ID and Secret from the WeCom admin console (stored in `.env`, masked previews, independent replace/clear per credential), manage the userid allow-list (display names in `config.json` are preserved), and optionally flip **Allow all organization members** — an explicit opt-in that lets everyone in your WeCom org tenant DM the bot without listing each userid (an empty allow-list still denies everyone; messages without a userid are always dropped). The status badge reports whether the channel actually started this session, with the failure reason when it didn't. diff --git a/NOTICE b/NOTICE index 2da32f2f876..45aa5e6b911 100644 --- a/NOTICE +++ b/NOTICE @@ -12,3 +12,11 @@ You may obtain a copy of the License at Portions of this software incorporate third-party components; see THIRD-PARTY-NOTICES for their respective copyright notices and licenses. + +The macOS computer-use feature's optional background-window click path +(src/kiro_crew/computer_use/macos_skylight.py) uses undocumented macOS +window-server interfaces. The symbol declarations and the event-field +recipe are derived from prior MIT-licensed open-source projects that +reverse-engineered that path. No third-party code is copied; the +implementation is independent, and the module is isolated so the +dependency is auditable in one place. diff --git a/config-baseline.json b/config-baseline.json index ca8841f25d2..ae88a0528db 100644 --- a/config-baseline.json +++ b/config-baseline.json @@ -1897,6 +1897,126 @@ "enumValues": null, "defaultValue": false }, + { + "path": "computer_use", + "kind": "core", + "type": "object", + "required": false, + "deprecated": false, + "sensitive": false, + "tags": [], + "label": "Computer Use", + "help": "Desktop automation tree/screenshot budgets. The primary enable is NOT here — it lives on the keystone computer_use.json.", + "hasChildren": true, + "enumValues": null, + "defaultValue": { + "max_tree_nodes": 1200, + "max_tree_depth": 64, + "text_limit": 500, + "attach_screenshot": true, + "screenshot_max_px": 1280, + "screenshot_jpeg_quality": 55, + "cursor_motion": false + } + }, + { + "path": "computer_use.max_tree_nodes", + "kind": "core", + "type": "integer", + "required": false, + "deprecated": false, + "sensitive": false, + "tags": [], + "label": "Max Tree Nodes", + "help": "Accessibility nodes one window walk may return before truncating.", + "hasChildren": false, + "enumValues": null, + "defaultValue": 1200 + }, + { + "path": "computer_use.max_tree_depth", + "kind": "core", + "type": "integer", + "required": false, + "deprecated": false, + "sensitive": false, + "tags": [], + "label": "Max Tree Depth", + "help": "How deep one accessibility walk descends.", + "hasChildren": false, + "enumValues": null, + "defaultValue": 64 + }, + { + "path": "computer_use.text_limit", + "kind": "core", + "type": "integer", + "required": false, + "deprecated": false, + "sensitive": false, + "tags": [], + "label": "Text Limit", + "help": "Characters kept per element title/value.", + "hasChildren": false, + "enumValues": null, + "defaultValue": 500 + }, + { + "path": "computer_use.attach_screenshot", + "kind": "core", + "type": "boolean", + "required": false, + "deprecated": false, + "sensitive": false, + "tags": [], + "label": "Attach Screenshots", + "help": "Capture the target window and relay the image path alongside the tree. The accessibility tree is always the primary channel.", + "hasChildren": false, + "enumValues": null, + "defaultValue": true + }, + { + "path": "computer_use.screenshot_max_px", + "kind": "core", + "type": "integer", + "required": false, + "deprecated": false, + "sensitive": false, + "tags": [], + "label": "Screenshot Width", + "help": "Longest edge of the downscaled screenshot, in pixels.", + "hasChildren": false, + "enumValues": null, + "defaultValue": 1280 + }, + { + "path": "computer_use.screenshot_jpeg_quality", + "kind": "core", + "type": "integer", + "required": false, + "deprecated": false, + "sensitive": false, + "tags": [], + "label": "Screenshot Quality", + "help": "JPEG quality 1-100 for the screenshot.", + "hasChildren": false, + "enumValues": null, + "defaultValue": 55 + }, + { + "path": "computer_use.cursor_motion", + "kind": "core", + "type": "boolean", + "required": false, + "deprecated": false, + "sensitive": false, + "tags": [], + "label": "Cursor Motion", + "help": "Draw a visible cursor gliding to each target before a real-pointer click, so the operator can see what the agent is doing. macOS only; purely visual and never a permit — the pointer opt-in and the governance permit are what allow the click itself.", + "hasChildren": false, + "enumValues": null, + "defaultValue": false + }, { "path": "mcp_gateway", "kind": "core", diff --git a/docs/system-specs/index.md b/docs/system-specs/index.md index 2791ad9fe7c..8b3c4dc17d8 100644 --- a/docs/system-specs/index.md +++ b/docs/system-specs/index.md @@ -14,6 +14,7 @@ Load relevant module specs before making changes to that component. Read common | [channel-history](modules/channel-history.md) | Group conversation context buffer | | [config](modules/config.md) | Dataclass config schema and loader | | [cli](modules/cli.md) | argparse CLI commands (chat, gateway, doctor, setup, manifest) | +| [computer-use](modules/computer-use.md) | Native desktop GUI automation: accessibility-tree snapshots, element-indexed actions, keystone primary enable, fail-closed in-gateway gate | | [heartbeat](modules/heartbeat.md) | Periodic background tasks | | [history](modules/history.md) | Persistent conversation history with LLM consolidation | | [knowledge](modules/knowledge.md) | Knowledge Library ingest (FileReader/SUPPORTED formats incl. .org), folder watcher, LLMPool workers (sweep-shielded) | diff --git a/docs/system-specs/modules/cli.md b/docs/system-specs/modules/cli.md index c44dc1012c2..95c90bae269 100644 --- a/docs/system-specs/modules/cli.md +++ b/docs/system-specs/modules/cli.md @@ -67,8 +67,13 @@ This allows `kirocrew` to find project-level agent config and skills from any di | `kirocrew config set --file ` | Replace config from a JSON file | | `kirocrew config edit` | Open config in `$EDITOR` | | `kirocrew memory show/edit` | Show or edit memory (preferences, projects, history) | +| `kirocrew computer doctor [--json]` | Report computer-use availability: platform support, the keystone primary-enable state, and the **advisory** macOS Accessibility / Screen Recording probe with a `responsible_hint`. See [Computer Use Commands](#computer-use-commands). | +| `kirocrew computer apps` | List on-screen applications the accessibility layer can address (human-facing twin of the `computer_list_apps` MCP tool). Gated by the same chokepoint as `call` — refused while the feature is off or the session is unattended. | +| `kirocrew computer call [k=v ...]` | Run ONE computer-use tool through the same gated chokepoint the agent uses, and print its reply (debug / reproduction) | +| `kirocrew computer call --calls '[…]'` | Run a JSON array of tool calls in a SINGLE process, so `element_index` values from an earlier `computer_get_state` are still resolvable | | `kirocrew mcp-cron` | MCP server for cron tools (spawned by kiro-cli) | | `kirocrew mcp-core` | MCP server for spawn, learn, task tools (spawned by kiro-cli) | +| `kirocrew mcp-computer` | MCP server for computer-use tools (spawned by kiro-cli; `argparse.SUPPRESS`-hidden). A **thin shim** — it forwards to the gateway over loopback and does no accessibility work itself. | | `kirocrew --version` | Print version | ## Token Command Output Streams @@ -659,3 +664,62 @@ dev-mode apps so the zero-dev-apps steady state costs one `stat()` per second. It is a derived cache reconciled from `installed.json` at watcher init (under a cross-process lock, atomic with concurrent toggles), **not** part of the App Kit contract — its path and format are internal and may change without notice. + +## Computer Use Commands + +`kirocrew computer {doctor [--json] | apps | call}` — hand-rolled dispatch +mirroring `browser/cli.py` (see [computer-use.md](computer-use.md)). + +**`doctor`** reports, in order: whether the platform is supported (macOS today; +Windows and Linux report a typed refusal), whether the keystone primary enable at +`~/.kiro/crew/computer_use.json` is on, and the macOS TCC probe +(`AXIsProcessTrusted()` + `CGPreflightScreenCaptureAccess()`). The probe is +**advisory and never a gate**: macOS attributes a grant to the *responsible +parent* of the process tree, so both rows can read `missing` while a +full-fidelity capture succeeds — observed live. `doctor` therefore prints a +`responsible_hint` naming the process a user should actually grant (the packaged +app, or the terminal that launched a dev gateway) and says outright that "not +detected" does not always mean unavailable. It never calls +`CGRequestScreenCaptureAccess`, which would pop a system dialog from a background +process. + +`--json` is the machine form the **gateway shells out to** for the Settings +permission rows. That indirection is deliberate: a short-lived subprocess keeps +native ctypes out of the gateway, so a native fault cannot take down the gateway +and with it cron, Slack and the dashboard WebSocket. + +**`apps`** lists on-screen applications resolved from +`CGWindowListCopyWindowInfo` (layer-0 windows only, never `pgrep` — a `pgrep -n` +lookup returns short-lived helper pids whose accessibility tree is empty). It runs +`computer_list_apps` through the SAME gated dispatcher as `call`, so it is refused +while the feature is disabled, in an unattended session, or under a policy that bans +computer use — the agent can run this command with bash, so an ungated version was +an unauthorized read of every window title. + +**`call`** runs one tool — `call computer_get_state app=Finder` — or a whole +sequence in ONE process: `call --calls '[{"tool":"computer_get_state","args": +{"app":"Finder"}},{"tool":"computer_click","args":{"app":"Finder", +"element_index":12}}]'`. The batch form exists because `element_index` values only +resolve against the per-process snapshot cache that produced them, so two separate +invocations cannot share them. `key=value` arguments are JSON-decoded when they can +be (`element_index=3` → int, `screenshot=false` → bool) and kept as text otherwise +(`app=Finder`). `--json` emits `[{tool, text}, …]`; the exit code is non-zero if any +reply carries the `Error: ` prefix, and a batch runs to completion rather than +aborting at the first refusal. + +`call` goes through `computer_use.tools.dispatch_tool`, the **same** chokepoint an +agent call traverses, so the primary enable, the target policy and the secure-field +floors all apply — it is a reproduction tool, not a bypass. Its session key is the +attended `cli_chat` surface, which is what the SEL audit records. There is no +separate diagnostics opt-in and no identity proof: the unattended-surface refusal +that made one necessary was removed along with the rest of the computer-use +governance model. + +All three are **human-facing**. `apps` has an MCP twin (`computer_list_apps`) per +the MCP-first rule; `doctor` is a permission diagnostic rather than a capability, +so the rule does not bind it; and `call` adds no capability at all — it is a +harness over the ten existing MCP tools, and deliberately has **no** MCP twin, +because a tool that runs other tools would let a model launder one per-call gate +decision into many. There is deliberately **no** `kirocrew computer state ` — +that would be a second, CLI-shaped spelling of an LLM-facing capability and would +have to be an MCP tool instead (it is: `computer_get_state`). diff --git a/docs/system-specs/modules/computer-use.md b/docs/system-specs/modules/computer-use.md new file mode 100644 index 00000000000..39ddabd6acc --- /dev/null +++ b/docs/system-specs/modules/computer-use.md @@ -0,0 +1,1514 @@ +# Computer Use Module (native desktop GUI automation) + +Lets the agent **read and drive the operator's own desktop applications** through +the platform accessibility layer: enumerate on-screen apps, walk one app window +into an indexed accessibility tree, then act on an element by index (press, +type, set a value, scroll, perform a named action) or — for UI that exposes no +addressable element — on a screen point (click, drag). macOS-only in this release; +Windows and Linux report a typed refusal. + +Two things this module is NOT, stated up front because both are load-bearing +product decisions rather than implementation gaps: + +- **Element-targeted, non-pointer input is the DEFAULT and the only thing + available out of the box.** `computer_click` with an `element_index` performs + `AXPress`, which activates a control with no pointer involved at all, and it is + what `auto` picks whenever an index is present. Coordinate clicking and + `computer_drag` are available for canvases, maps and custom-drawn UI, and they + too leave the physical cursor alone — `click_method: "app_post"` posts a + *located* mouse event to the target process with `CGEventPostToPid`. + + The one path that warps the real cursor is `click_method: "global"`. It has to be + NAMED by the model — `auto` never resolves onto it — and every use emits a SEL + record under its own `tool_kind`, so the pointer stays where the operator left it + unless the model explicitly asked otherwise. Earlier revisions of this document + asserted flatly that "the pointer never moves"; the accurate statement is that it + never moves *by accident*. See + [Coordinate clicking, drag and the real-pointer path](#coordinate-clicking-drag-and-the-real-pointer-path). +- **It is off until the operator turns it on**, out-of-band, in a file the agent + can neither read nor write. See [The keystone primary enable](#the-keystone-primary-enable). + +Two optional, human-facing views ride alongside, neither of which grants the agent +anything: the [live view (PiP)](#the-live-view-pip) mirrors screenshots the model +already read, and [Cursor Motion](#cursor-motion-a-real-desktop-overlay-cosmetic-only) +draws a fake cursor on the real desktop so a watching human can see where a click is +about to land. Cursor Motion is **not** the pointer and is deliberately invisible to +`screencapture`. + +> Prior art: the tool surface and the per-turn element-index discipline are +> modelled on the open-source `open-codex-computer-use` MCP contract (MIT), +> which we probed to validate the shape. No code is derived from it — the driver, +> the compression pipeline and the security floors are all KiroCrew's own. + +--- + +## Architecture: thin shim, in-gateway dispatch + +``` +kiro-cli + └─ spawns kirocrew mcp-computer (stdio MCP server: kirocrew-computer) + │ THIN SHIM — resolves session identity strictly, forwards, returns text + ▼ + POST /api/computer-use/invoke (loopback, X-Internal-Secret) + │ + ▼ GATEWAY PROCESS + dashboard/handlers/computer_use.py + └─ computer_use/service.py ── the ONE dispatch chokepoint + 1. enable_state.is_enabled() keystone primary enable + 2. computer_use/policy.py::check_app target policy (self + operator lists) + 3. index freshness (TTL) + fingerprint re-walk + 4. policy.check_input_target secure field / text scan + 5. computer_use/gate.py::require_computer_use SEL audit (no decision) + 6. ComputerUseBackend ──► macos_driver → macos_ffi (ctypes) + 7. re-snapshot, policy.redact_result +``` + +### Why the stdio process is a thin shim and the work happens in the gateway + +The MCP stdio process does **no** accessibility work and **no** auditing. It checks +the keystone enable (so a disabled feature advertises zero tools), POSTs to the +gateway over loopback with the `X-Internal-Secret` handshake (the pattern +`mcp_core.py` already uses — the same `.local_secret` file; `X-Local-Secret` is its +sibling header, read only by `GET /api/token/local`), and relays the text result +back. Everything of consequence happens in the gateway. + +Three reasons the split is worth the extra hop, none of them governance: + +* **the native work must not run in the shim.** A ctypes fault is not catchable in + Python. In the gateway it is contained by the driver's `_guarded` seam and the + bounded `subprocess_executor` pool; in a short-lived stdio child it would take the + child down mid-call with no result to relay; +* **the snapshot cache has to be shared.** Element indices only mean anything + against the walk that produced them, and that cache (`index.SnapshotIndex`) lives + in the gateway. A per-shim cache would make every follow-up action refuse; +* **the audit belongs where the action happens.** SEL is a gateway service, and the + trail is now the operator's primary record of what the agent did to their desktop. + +**The shim MUST be told the data home.** Both processes read the keystone +`computer_use.json`, and a child does **not** inherit the gateway's +`KIROCREW_HOME` — the managed spec's `env` map is the only channel, so +`agent._managed_mcp_env()` pins it there (for every managed server, not just this +one). Without the pin the two sides read DIFFERENT homes, and the failure mode is +worse than a plain error because it is silent and self-contradictory: Settings writes +`enabled: true` to the override home, `mcp_computer` reads `false` from the default +one and publishes an empty `tools/list`, so the panel shows the feature ON while the +agent truthfully reports it has no computer-use tools. Both are telling the truth +about different files. Found by running the feature under `KIROCREW_HOME` in dev mode. + +The pin is resolved through `paths._valid_override_home()` rather than reading the env +var, so an override the loader itself REFUSES is not handed to a child that would then +disagree in the other direction — the guarantee is *agreement with the gateway*, not +validity. It is refreshed like `command`/`args` (not preserved like `autoApprove`, +which is a user customization): a config written under an override and later refreshed +on a default install has the stale key REMOVED. A default install emits no `env` at +all, so the spec is byte-for-byte unchanged there. Pinned by +`test_computer_use_registration.py::TestDataHomePin`. + +**Session identity.** The shim resolves it with +`mcp_core._resolve_session_key_strict()` — the env `KIROCREW_SESSION_KEY`, else +`KIROCREW_HOST_PID` plus the HMAC sidecar signed with the keystone-protected +`sel_hmac.key`. It is used for the audit record and for the live-view relay's +attribution, **not** as an authorization input: an unresolved key does not refuse the +call, because there is no per-surface ceiling left for it to select. + +That is enforced in the SHIM as well as the gateway, and it has to be: neither +accepted source exists for a GUI-launched kiro-cli on **macOS**, the only platform +with a driver. `KIROCREW_SESSION_KEY` is injected only by the ACP spawn path +(`acp/client.py`), and `KIROCREW_HOST_PID` only by the Linux sandbox launcher +(`sandbox.py:666`). An earlier revision refused in the shim on the reasoning that an +unproven key is indistinguishable from an unattended surface — with the unattended +rule gone, that left the feature returning *"the calling session could not be +identified"* for every ordinary dashboard chat on macOS. Found by using it. + +The STRICT resolver is still the one called: the lenient variant walks a file +`mcp_core` documents as "agent-writable and therefore forgeable", and an empty audit +identity is honest where a forged one is a lie. The cost is attribution, not a +control. `test_mcp_computer.py::test_the_shim_carries_no_identity_refusal_at_all` +guards the absence, because a behavioural test alone passes just as well with a +refusal that happens to be unreachable. + +**An unresolved key becomes `unresolved:`, never the empty string** — and +that is a correctness fix, not cosmetics. `SnapshotIndex` namespaces entries by +`(session_key, window_key)`, so an empty key collapsed EVERY unresolved session onto +one `("", window)` slot. Since unresolved is the normal case on macOS, two concurrent +sessions observing the same window overwrote each other's element indices — and each +one's own `verify_fingerprint` still passed, because both trees describe the same +window, so the wrong-target action had nothing reporting it. kiro-cli spawns one shim +per session, so the shim's own pid separates the namespaces exactly as far as the +sessions are genuinely separate. Read at call time rather than captured at import, so +a forked child cannot inherit its parent's string and re-alias with it. + +The prefix is deliberate: this is a namespace separator, not attribution, and an audit +reader must not mistake a pid for a resolved identity. And it is a namespacing fix +specifically **because** the alternative — refusing an empty key — is the line that +made the feature unusable on macOS; the security posture is unchanged, only the cache +key is. + +### Why a separate MCP server rather than folding into `kirocrew-core` + +`config/defaults.json` blanket-allowlists `@kirocrew-core` in `allowedTools`. +Riding inside it would inherit that blanket auto-approve for `computer_click`. A +separate slash-free server key also lets a fleet deny `@kirocrew-computer` with +one `mcp`-scope pattern. `@kirocrew-computer` is added to `tools` but +deliberately **not** to `allowedTools`, and the managed server spec carries **no +`autoApprove` key** — an autoApproved MCP tool is approved locally by kiro-cli, +emits no permission request, and never reaches `hooks.on_tool_call`. + +### The backend seam and the shipped fake + +`ComputerUseBackend` (`backend.py`) is a plain `abc.ABC` with a runtime-swappable +factory (`register_computer_use_backend` / `get_shared_backend` / +`reset_shared_backend`) — the shape `embeddings.register_embedding_backend` +already uses. It is deliberately **not** a `PlatformContext` extension point: +CPP is the *edition* seam (standalone vs companion) and its `CONTRACT_VERSION` is +pinned at 1, while computer use varies by *operating system*; a registry also +stays swappable inside a single pytest process. CPP still owns +`redact_via_context` (so a loaded companion's extra credential patterns apply to +computer-use output); the ABC owns **who acts**. + +Contract every implementation honours: never raise (every failure becomes +`DriverResult(ok=False, text=…)`), **never move the pointer unless the request +says to** (only a `ClickRequest`/`DragRequest` whose `moves_pointer` is True, which +the chokepoint only builds after both permits cleared — a driver must never +upgrade a method on its own), set `ElementRec.secure` from BOTH role and subrole, +stamp `Snapshot.captured_at` from `time.monotonic()`, be thread-safe. `UnsupportedBackend` is a concrete shared base whose every method +returns the same typed refusal, so a new unsupported platform is ~10 lines and +cannot accidentally implement half a driver (`windows_driver.py`, +`linux_driver.py`). + +`kiro_crew/testing/fake_computer_use.py` ships a `FakeComputerUseBackend` in the +runtime wheel (alongside `fake_acp_backend.py`), so a downstream suite can drive +the whole stack with no framework, no window, no application and no permission +grant. Its fixtures exist to make the security branches *reachable*: a node with +`role="AXTextField"` + `subrole="AXSecureTextField"` and a populated value, a +node whose title carries a credential-shaped literal and an exfil-shaped URL, a +blocked (terminal) app in the catalog, and a real decodable 1x1 JPEG. The suite +registers it process-wide; combined with the structural guarantee that no module +in the package calls `CDLL` at import scope, CI can never touch the native path. + +Importing `kiro_crew.computer_use` is side-effect free: no framework load, no +file read, no platform branch until `get_shared_backend()` is called. +`select_default_backend()` is the ONLY platform branch in the package and it asks +`platform_compat.IS_MACOS` / `IS_WINDOWS` / `IS_LINUX`, never `sys.platform` — +which is also what lets a Linux runner exercise the Windows degradation path by +flipping one flag. + +--- + +## The 10-tool contract + +Server `kirocrew-computer` (slash-free: kiro-cli splits an agent `@server` +reference on `/`). All tools prefixed `computer_` so they can never collide with +the playwright server's `browser_*`. + +| Tool | Required | Optional | Class | +|---|---|---|---| +| `computer_list_apps` | — | — | observe | +| `computer_get_state` | `app` | `text_limit` (1..20000, d=500), `max_tree_nodes` (1..5000, d=1200), `max_tree_depth` (1..128, d=64), `screenshot` (bool, d from config) | observe | +| `computer_click` | `app` + **exactly one of** (`element_index` \| `x`+`y`) | `click_count` (1..3, d=1), `mouse_button` (`left`\|`right`\|`middle`, d=left), `click_method` (`auto`\|`accessibility`\|`app_post`\|`sky_click`\|`global`, d=auto) | mutate, pointer | +| `computer_drag` | `app`, `from_x`, `from_y`, `to_x`, `to_y` | `mouse_button`, `click_method` | mutate, pointer | +| `computer_type_text` | `app`, `text` (≤10000), `element_index` | — | mutate, keyboard, text_entry | +| `computer_press_key` | `app`, `key` (≤64), `element_index` | — | mutate, keyboard | +| `computer_set_value` | `app`, `element_index`, `value` (≤10000) | — | mutate, text_entry | +| `computer_scroll` | `app`, `element_index`, `direction` (`up`\|`down`\|`left`\|`right`) | `pages` (0.1..20, d=1.0) | mutate, pointer | +| `computer_perform_action` | `app`, `element_index`, `action` (≤64) | — | mutate, pointer | +| `computer_end_turn` | — | — | control | + +`computer_click`'s "exactly one of" is a CROSS-FIELD rule, so it is **not** in the +schema (`validate_tool_args` checks fields independently and has no vocabulary for +it) — it is enforced at the dispatch chokepoint by `policy.check_click_target`, +which the in-process entry point also traverses. Both failure modes are refused +rather than resolved by precedence: silently preferring the index would make a +model that meant the coordinates act somewhere else entirely, in a live +application, with no signal that it happened. + +**`element_index` is REQUIRED on both keyboard tools, and that is a security control +rather than an ergonomic choice.** There is no "type into whatever is focused" form: +an unnamed target has no role or subrole, so `policy.check_input_target`'s always-on +secure-field refusal has nothing to inspect, and an indexless keystroke would type +into a focused password box. `computer_press_key` is included for a second reason — +`press_key("tab")` can *move* focus onto a password field, and the following keystroke +would land there. `computer_click` is the only element-scoped tool that accepts an +alternative, and only because coordinates are a target it can check +(`policy.check_click_target`'s one-of). Enforced by `_ELEMENT_REQUIRED_TOOLS` at the +chokepoint; `SKILL.md`'s tool table states the same thing, since an optional-looking +argument there would have the model discover the refusal by hitting it. + +Every tool has a `MCP_COMPUTER_SCHEMAS` entry in `validation.py`. That is +mandatory, not tidiness: an unregistered tool's arguments pass RAW through +`_validate_args`, and a `ValidationError` raised inside a handler escapes the +stdio loop and kills the server. + +`computer_list_apps` and `computer_get_state` are the observation tools; +`computer_end_turn` is control-plane (it drops KiroCrew's *own* cached snapshots +and touches no other application, so it is neither observe nor mutate). The +class labels above are the code-owned `governance._CU_ACTION_CLASSES` table — +see [governance.md](governance.md). + +Both spool writers (`service._persist_image` and `capture_macos.persist_jpeg`) name +their files with `tempfile.mkstemp`, not a millisecond timestamp. `_shot_lock` +serializes writers within one service instance but cannot serialize a second +PROCESS — the gateway, the CLI and the permission-probe child all spool into the +same `tempfile.gettempdir()` directory — so a timestamp-only name let two captures +in the same millisecond resolve to one path, the second truncating the first and +leaving its caller holding a screenshot of an application it never asked about +(a cross-capture pixel leak, reviewer finding). `mkstemp` also creates the file +`0o600` from the outset, so there is no window in which it exists world-readable +before `restrict_to_owner` runs. The timestamp stays in the *prefix* because the +ring trim orders by name. + +**`computer_list_apps` only omits an app it cannot name.** It used to carry a +per-app governance filter (`gate.app_is_disclosable` against the `computer_use.apps` +/ `.app_names` axes) because it is the one verb that names every application and +resolves no target, so `require_computer_use` could not see them. With those axes +gone the function survives as a shape check: an app with neither a bundle id nor a +display name is dropped because there is nothing to show. Terminals and password +managers now appear in the list like everything else. + +### Result shape: text only, by construction + +Exactly what `validation.build_tool_response` emits: + +```json +{"content": [{"type": "text", "text": "…"}]} +``` + +Text only, capped at `MAX_RESPONSE_LEN`. **There is no `isError` field and no +image block** — an image block is not expressible on this transport, so +"tree-first, relay the screenshot as a path" is a property of the transport +rather than a policy someone can regress. An error is the literal string +`"Error: …"`; `mcp_shared.call_tool_with_logging` classifies that prefix as SEL +`outcome="failed"`, so the prefix is load-bearing. + +Rendered body: + +``` +App=com.apple.finder (pid 1041) +Window: "Documents", App: Finder. + +0 window "Documents" + 1 splitgroup + 2 scrollarea + 3 button "Back" [AXPress] + 7 textfield +[tree truncated at 1200 nodes] + +Screenshot: /var/folders/…/kirocrew-computer-shots/shot-1769472013411.jpeg + (1280x604 jpeg, 24.2 KB) — read it with the fs_read tool only if the tree is + insufficient. +``` + +Every mutating tool returns the REFRESHED tree at the configured budgets, so the +model always acts against indices it has just been shown. Its response is +`"\n\nRefreshed state:\n"`, and **the two halves are redacted +separately — deliberately, and neither is optional.** + +The tree half is redacted inside `render_tree`; the header was concatenated *after* +that pass, and `detail` is not our prose — every driver confirmation interpolates +app-supplied text (`_click_text` embeds `app.name`, the process name macOS reports, +which is attacker-controlled). A process named `Notes key=AKIA…` therefore put a raw +credential directly in front of a fully redacted tree. Fixed by redacting `detail` at +the interpolation. + +Redacting the *joined* string instead would break screenshots: `render_tree` appends +its image note after its own redaction because the per-user temp dir contains a long +random segment that `redact_credentials`' bare-secret-key heuristic masks, so a second +pass would replace every screenshot path with a placeholder and the channel would +silently never work (verified live; see `render._render_image_note`). Hence: header +redacted on its own, already-redacted body passed through untouched. Both halves of +that rule are pinned by tests, the second one structurally — a mutating tool's refresh +walk is `want_image=False`, so no behavioural test in that file would notice a +"just redact the whole response" simplification breaking the read path. + +--- + +## The keystone primary enable + +The primary enable lives at **`~/.kiro/crew/computer_use.json`**, NOT in +`config.json`: + +```json +{ "enabled": false, "allowed_apps": [], "extra_denied_apps": [] } +``` + +Why not `config.json` — verified, and with a precedent in this repo: +`is_sensitive_write_path("~/.kiro/crew/config.json")` is `True` (the tool path is +protected) but `is_sensitive_bash_command("echo x > ~/.kiro/crew/config.json")` is +`None` and `is_denied(...)` is `None`. `security.py` states the governing +precedent outright: the denied-command opt-out is deliberately kept OFF +`config.json` **because it is a security ceiling**. A primary enable for full +desktop observation plus input synthesis is the same class of control, so +`computer_use.json` is added to `security._CREW_SECRET_LEAVES`, which gets +read+write protection on both the tool path (`is_sensitive_path`) and the shell +forms (`is_sensitive_bash_command`, including `cat`, `>`, `tee`, and +`tar -C`/`unzip -d` extraction into the trust root). + +Mechanics: + +- Every read fails soft to `{}` → **DISABLED**. Absent, unreadable, truncated or + hand-mangled must never mean "enabled". +- `is_enabled()` is strict identity against `True`: a hand-edited + `"enabled": "false"` (a truthy string) or `"enabled": 1` does not enable + desktop control. +- The only writer is the dashboard PUT handler, which does not route through the + agent tool gate. +- `allowed_apps` is an optional narrowing (empty = everything not denied); + `extra_denied_apps` can only ADD. There is deliberately no mechanism to remove + a built-in denylist entry. + +`config.json`'s `computer_use` section carries **display and limits only** — +`max_tree_nodes`, `max_tree_depth`, `text_limit`, `attach_screenshot`, +`screenshot_max_px`, `screenshot_jpeg_quality`. The absence of an `enabled` field +there is deliberate; see [config.md](config.md). + +--- + +## HTTP surface + +Four routes in `dashboard/handlers/computer_use.py`, and the split in their auth +models is the point. + +| Route | Auth | Caller | +|---|---|---| +| `GET /api/computer-use/config` | cookie (browser) | Settings panel | +| `PUT /api/computer-use/config` | cookie (browser) | Settings panel | +| `POST /api/computer-use/invoke` | loopback + `X-Internal-Secret` | the stdio shim ONLY | +| `POST /api/computer-use/frame` | loopback + `X-Internal-Secret` | this gateway's own capture thread ONLY | + +`invoke` is in `server._STRICT_INTERNAL_API_PATHS` — no cookie fall-through, and +non-loopback is denied outright. It is the entry point to accessibility reads and +input synthesis, so it is the one route where a cookie fall-through would be a +genuinely new attack path rather than a convenience. It is registered in +`_register_mcp_routes` so the headless `--slack-only` server exposes it too (kiro-cli +spawns the shim on both entrypoints). The config pair is deliberately NOT in that +set: it is browser-called, like the browser-config pair. + +**`GET /api/computer-use/config` fails SOFT on a malformed keystone, and that is a +deliberate inversion of the action path.** `load_policy_config` raises +`PolicyStateError` on a present-but-malformed `allowed_apps` because coercing it to +the empty tuple would silently convert an operator's restriction into no restriction — +right for a dispatch, wrong for a read. Letting it escape `_snapshot()` turned the GET +into an HTTP 500, so a hand-edited keystone made *the only UI that can repair the +file* unreachable; the page has to render precisely because the file is broken. The +handler therefore falls back to an empty `PolicyConfig` and publishes `policy_error`, +which the panel renders as a warning naming the file — an empty allow-list otherwise +reads as "no restriction configured", the opposite of what the operator wrote. **The +ceiling is unchanged:** every dispatch still calls `load_policy_config` itself and +still refuses on the same value, so only the *rendering* degrades. A test asserts both +halves (the GET renders; the action path still raises). + +`frame` is in the strict set for the same reason and registered in the same block +(a `--slack-only` gateway drives the desktop too; it simply has no owner sockets +to deliver to). Like `invoke`, it re-asserts BOTH the loopback check and +`request["internal_auth"]` inside the handler: the strict listing does not prove +the secret was checked (an absent header falls through to cookie auth, and +`local_only=False` reclassifies every strict path as "mixed"), so without that +assertion a caller holding only a dashboard cookie or an app-scoped token could +inject arbitrary frames into every owner window's live view. See +[the live view](#the-live-view-pip) below. + +**`GET /api/computer-use/config`** returns `{enabled, supported, platform, reason, +max_tree_nodes, max_tree_depth, text_limit, attach_screenshot, screenshot_max_px, +screenshot_jpeg_quality, allowed_apps, +extra_denied_apps, permissions{accessibility, +screen_recording, responsible_hint}, limits{field: [min, max]}}`. + +- `permissions` comes from shelling `kirocrew computer doctor --json` + (`asyncio.create_subprocess_exec`, fixed argv, 5s timeout, one + `test_spawn_audit.BENIGN_SPAWNS` entry). Degrades to `unknown` on timeout, + non-zero exit or unparseable output, and reports `unsupported` off macOS without + spawning at all. A timed-out child is killed — the panel polls every 5s while a + grant is outstanding, so leaking one per poll would pile up. +- `limits` publishes the server's own ceilings so the panel's number inputs bound + themselves rather than re-spelling them in TypeScript. + +**`PUT /api/computer-use/config`** accepts any subset of `{enabled, allowed_apps, +extra_denied_apps}` (→ keystone) and the six budget fields (→ `config.json`), and +returns the refreshed GET payload. Everything is validated before anything is +written. `409` (matching `api_denied_command_builtin_toggle`, not the runtime +chokepoints' `403`) when the ceiling forbids and the request would **widen** the +surface — enabling, or editing the target lists. Disabling and shrinking a budget +are always allowed: narrowing cannot conflict with a ceiling, and a user must +never be locked out of switching this off. A corrupt keystone or `config.json` is +`500` and is left byte-identical rather than clobbered +(`StateCorruptError`, the `ConfigCorruptError` precedent). Every mutation SEL-audits +the decision (`enabled=…`, changed field names) — never the app patterns +themselves. + +**`POST /api/computer-use/invoke`** takes `{tool, args, session_key, agent, app}` +and returns `{"text": …}` with a 200 for BOTH success and refusal, because a +computer-use refusal is a tool result (`"Error: …"`, which the SEL layer classifies +as failed) rather than a transport failure the model cannot reason about. Only a +malformed request gets a 4xx. The identity fields are not an authorization claim +this handler trusts — the shim resolved them strictly, and the fail-closed gate +treats an empty `session_key` as unattended and denies; the handler never infers +one. The dispatch runs in a worker thread (accessibility calls block for tens of +milliseconds). + +--- + +## What is enforced, and what is not + +One operator opt-in, then the agent drives the desktop the way the operator would. +That is a deliberate product decision and it replaced a much larger governance +model — eight `SCOPE_CATALOG` rows, an unattended-surface refusal, an +interactive-approval floor, a per-app disclosure filter and a five-category app +denylist. Those are **gone**; this section is the honest replacement. + +### The dispatch chokepoint, in order + +`tools.py::_dispatch` is still the single funnel every tool passes through, and the +order still matters, but there are only four steps left: + +1. **Schema validation** — `validation.MCP_COMPUTER_SCHEMAS`. An unregistered tool + is refused before anything else, because unvalidated arguments reaching a handler + would escape the stdio loop and kill the server. +2. **The keystone primary enable** — `enable_state.is_enabled()`. One read serves + both the enable test and the operator's target lists, so a hand-edited file + cannot be observed in two states within one dispatch. +3. **OS identity resolution** — the window list only. The app the driver *resolved* + is what gets audited, never the agent-supplied `app` string. +4. **The target policy** — `policy.check_app` (the one retained denylist entry plus + the operator's own allow/deny lists) and `policy.check_input_target` (the + secure-field refusal). + +`gate.require_computer_use` still sits at step 4's boundary and still has the same +signature, but it now only **audits**: there is no governance decision left to make. +Keeping the call (rather than deleting it) is what keeps the audit trail complete +and gives a downstream edition one place to reintroduce a decision. + +### What still refuses + +| Refusal | Where | Why it survived | +|---|---|---| +| Feature off | `enable_state` + `tools` step 2 | The keystone is on `security._SENSITIVE_HOME_DIRS`, so the agent can neither read nor write it. This is the whole security model now. | +| KiroCrew's own window | `policy.check_app` | Driving our own Settings UI would let the agent click the enable above — a self-escalation loop that would make the keystone pointless. Matched on bundle id, process name **and window title** — see below. | +| Password fields | `policy.check_input_target` + `render` + `capture_macos` | A privacy floor, not a policy knob: the value is never read, never rendered, and a window holding one is never photographed. | +| Operator's own app lists | `policy.check_app` | `allowed_apps` / `extra_denied_apps` on the keystone. The operator's choice, not a shipped ceiling. | +| Stale / drifted element index | `index` + `service.verify_fingerprint` | Correctness, not authorization — acting on a stale index clicks the wrong control. | +| Credential-shaped output | `policy.redact_result` | The repo-wide egress control every other surface already runs. Applied to the action HEADER as well as the tree — the header interpolates app-supplied text (a process name), so it was an unredacted egress path on its own. | +| Indexless keyboard input | `tools._ELEMENT_REQUIRED_TOOLS` | `computer_type_text` / `computer_press_key` REQUIRE `element_index`. An unnamed target has no role or subrole, so the password-field check above has nothing to inspect and the keystroke would land in whatever the app happened to have focused. `press_key("tab")` is included because it can *move* focus onto a password box. An earlier draft of this document listed this under "no longer refuses"; that was never implemented, and the doc was the thing that was wrong. | +| Non-left `sky_click` | `policy.check_method_button` | The private recipe is a left-button sequence. Refused rather than downgraded, because synthesizing a left click for a right-click request performs a different gesture than the one asked for. | + +### What no longer refuses + +Stated plainly, because these are behaviour changes a reader will otherwise trip +over: + +* **unattended surfaces** — cron, subagent, taskrunner, webhook, workflow and + channel sessions all drive the desktop. There is no `UNATTENDED_SURFACES` rule; +* **terminals, password managers, System Settings and system auth dialogs** — all + readable and drivable. The shipped denylist that covered them was incomplete by + construction (an IDE's embedded terminal was never matched) and got in the + operator's way on their own machine; +* **the real-pointer path** — `click_method: "global"` needs no second opt-in and no + governance permit. It still has to be NAMED by the model (`auto` never resolves to + it), so the cursor is never warped by accident, and every use is audited under its + own `tool_kind`; +* **paste** — `cmd+v` is allowed; +* **observation channels** — `apply_observation_ceiling` is a pass-through. Window + titles, element values and file paths are not narrowed; +* **interactive approval** — there is no `computer_use.approval` row, so nothing + makes the feature observation-only. + +### Accountability replaces authorization + +With the ceiling gone, the SEL audit trail is what the operator has. Every call is +recorded (`gate._audit_allowed`), every refusal is recorded (`tools._refusal` / +`_static_refusal`), and a real-pointer gesture gets its own `tool_kind` +(`computer_use_pointer`) so "did the agent ever take control of my mouse?" is one +filter over the log rather than a parse of every row. + +## Coordinate clicking, drag and the real-pointer path + +Element addressing is the preferred path and stays the default: `AXPress` activates +a control with no pointer at all, needs no pixel measurement, and is what `auto` +picks whenever an `element_index` is present. But some UI has no addressable +element — canvases, maps, timelines, custom-drawn controls — and some gestures have +no accessibility form at all. Hence a coordinate `computer_click` and a +`computer_drag`. + +### The click methods + +| Method | Delivery | Moves the cursor? | Buttons | Needs | +|---|---|---|---|---| +| `accessibility` | `AXUIElementPerformAction(elem, "AXPress")` + the ladder, then the enclosing control | no | left (press ladder), right (`AXShowMenu` only) | `element_index` | +| `app_post` | `CGEventCreateMouseEvent` + `CGEventPostToPid` | **no** — verified live: the prototype's cursor position was identical before and after | all | `x`+`y` | +| `sky_click` | private SkyLight recipe (see below) | **no** | **left only** | `x`+`y`, a resolved window id | +| `global` | `CGWarpMouseCursorPosition` + `CGEventPost(kCGHIDEventTap)` | **YES** | all | `x`+`y`, and the model must NAME the method | +| `auto` (default) | resolves to `accessibility` when an index was given, else `app_post` | no | all | — | + +**`auto` never resolves to `global` or to `sky_click`.** That is an invariant with +tests, not a preference: an implicit resolution onto the pointer-warping path would +let a model take the operator's mouse without ever naming the method, and an implicit +resolution onto the private path would put undocumented ABI on the default route. + +**`sky_click` is left-button only, and a non-left request is REFUSED rather than +downgraded** (`ERR_SKY_CLICK_BUTTON`, gated at the chokepoint by +`policy.check_method_button` and re-checked inside `macos_skylight`). The recipe was +reverse-engineered for a left click and the button number is one field among nine, so +a right-click variant would be invented rather than observed. Downgrading was the +actual bug an earlier revision shipped: the recipe took no button at all and built +left-button codes unconditionally, so a right-click request silently *activated the +control* instead of opening its context menu — on a background window the operator +cannot see. Same reasoning as `AX_MENU_LADDER` never falling back to `AXPress`: +performing a different gesture than the one requested is worse than performing none. + +### `sky_click` — the private path, and why it IS shipped + +`sky_click` clicks a window that is **behind other windows**, without raising it and +without moving the pointer. It is the only method built on undocumented Apple ABI, +and an earlier revision of this feature deliberately did NOT port it on the grounds +that a shipped product should not depend on an API Apple can remove. + +**What reversed that.** The gap is real and reachable: a canvas window covered by +another app's overlay cannot be clicked by ANY public method. `accessibility` needs +an addressable element (a canvas has none), and `app_post` is delivered to the app +but ignored by Chromium- and Catalyst-based renderers, which hit-test against the +window server's idea of what is in front. Hit in practice on a Freeform canvas behind +a Zoom annotation overlay: every public method refused or silently did nothing. + +**How the trade is contained**, rather than accepted wholesale: + +- the symbol declarations and event recipe come from prior permissively-licensed + open-source reverse-engineering work, attributed in `NOTICE` rather than in code + comments (no third-party code is copied — the implementation is independent); +- the private ABI lives in ONE module, `macos_skylight.py`, so a macOS point release + that changes a byte offset has one review boundary. `macos_ffi.py` keeps its + "public frameworks only" property, which is what makes it reviewable against + Apple's documentation. `test_computer_use_skylight.py` asserts that quarantine + structurally — the private symbols must not appear in any other module; +- it **fails closed with a readable refusal.** `available()` reports which symbols + are missing, and every entry point refuses in prose naming `app_post`. A future + macOS that drops `SLEventPostToPid` costs the model one clear refusal, never a + crash and never a mis-delivered click; +- it is reachable **only by name.** `auto` never resolves onto it, exactly as with + `global` — a model that did not ask for a private-API path never gets one; +- the **byte layout and event order are pinned by tests**, because they are the parts + an edit can change without a linter or type checker noticing, and the failure mode + is not an exception: it is a click delivered to the wrong window. The primer + down/up pair at `(-1, -1)` is asserted present for the same reason — it is what + routes the real click to the target rather than to whatever is frontmost. + +Note the shape other shipped implementations of this technique have converged on: at +least one isolates the private surface in a **separate signed helper process** it can +update or revoke independently of the main binary. Quarantining to a module is the +same instinct one level less strict — worth revisiting if this surface grows, since a +process boundary also bounds a crash, not just a review. + +### Audit + +Every pointer-moving action emits a SEL record naming the METHOD — +`gate.audit_pointer_move`, `tool_kind="computer_use_pointer"` — on the allow path as +well as the deny path, and *in addition to* the ordinary `tool_invocation` +record. The generic record cannot answer "did the agent ever take control of my +mouse?" because a pointer-moving click is indistinguishable from an `AXPress` in it, +and that is the one question this path exists to keep answerable. + +### FFI notes (each cost a debugging cycle) + +- `CGEventCreateMouseEvent` takes a `CGPoint` **by value**. Two bare doubles have + the same total size but a different AArch64 register layout, so the *button* + argument lands in the wrong register. +- There is **no generic "mouse down"** — the event type is PER-BUTTON. A right-click + posted as `kCGEventLeftMouseDown` with `button=1` is delivered as a LEFT click. + `macos_ffi.MOUSE_EVENT_TYPES` carries a distinct down/up/dragged triple per button. +- A double click is **not** two pairs: it is a pair whose `kCGMouseEventClickState` + is 2, which is where AppKit reads `NSEvent.clickCount` from. +- A drag needs the intermediate `MouseDragged` events (`DRAG_STEPS = 6`) and a small + per-step delay. A bare down/up pair is not a drag to most apps — the gesture is + recognized from the motion, and identical timestamps let the recognizer coalesce + the sequence. TextEdit selected nothing without both. +- Mouse events use the same PRIVATE event source and the same explicit + `CGEventSetFlags(event, 0)` as keystrokes: the modifier hygiene is about the source + of the event, not about whether it is a keystroke. +- `CGEventPost` and `CGWarpMouseCursorPosition` are called from exactly two + functions (`post_mouse_global`, `post_mouse_drag_global`), and a test pins that + call-site set — a new caller would be a new ungated path to the operator's mouse. + +--- + +## What one accessibility walk reads (and why each field is worth its round-trip) + +The tree is the primary channel, so every field it omits is a turn the model spends +recovering the information some other way — usually by guessing a coordinate and +reading back a screenshot. These are the reads that pay for themselves. + +**Element frames** (`ElementRec.frame`, from `AXPosition` + `AXSize`). Both come +back as **`AXValue` boxes**, not plain numbers, so each is type-checked with +`AXValueGetType` before `AXValueGetValue` unboxes it into the matching struct — a +`CGPoint` read into a `CGSize` would transpose `y` into `width` and produce a +plausible-looking rect pointing somewhere else, which is strictly worse than no +rect. A half-read (position without size) yields `None` rather than a partial +rectangle for the same reason. + +Frames are **window-local**, and `Snapshot.window_bounds` publishes the origin they +are relative to. Three reasons, in order of how badly the alternative fails: + +1. the screenshot the model may also be reading is a **crop of the window**, so a + screen-absolute rect could not be related to any pixel it can see; +2. a window-local frame survives the user dragging the window between turns; +3. an unlabelled coordinate is unusable — a consumer cannot tell window-local + `(12, 40)` from screen-absolute, and the difference is the window's position. + +Because `computer_click(x, y)` takes SCREEN coordinates, the rendered origin line +states the conversion explicitly rather than leaving the model to infer that two +coordinate systems are in play. Without the origin, a frame passed straight to a +coordinate click lands off by the window's position — and on a maximised window it +would appear to work, which is the worst way for this to fail. + +**Traits** (`selected`, `expanded`, `editable`). Read as a **tri-state**: only a +definite `True` renders a word, because `AXSelected` is unsupported on the great +majority of elements and collapsing absent into `False` would attach a trait to +every node in the tree. `editable` is the load-bearing one and comes from +`AXUIElementIsAttributeSettable(AXValue)`, not from `AXEnabled` — those answer +different questions. A read-only text field (a disabled form input, a log pane, a +computed cell) reports `AXEnabled=true` with a readable value, so it is +indistinguishable from a writable one in the tree: the model types into it, gets an +`ok` result, and the text silently goes nowhere. Settability is the only signal that +separates them, and it turns that dead end into something visible *before* acting. + +**Focus and selection** (`ElementRec.focused`, `Snapshot.selected_text`). Both are +read ONCE per walk off the **application** element, not per node and not +system-wide. Per-node would add a round-trip to each of up to ~1,400 nodes to answer +a question with one answer. System-wide focus follows whatever the *operator* is +working in, so it would mark a background app's element whenever the target happened +to be frontmost and report nothing the rest of the time; the app-scoped attribute +answers "where is this app's caret", including for a background app. Identity is +compared with `CFEqual`, never pointer equality: `AXFocusedUIElement` is a +Copy-Rule read returning a fresh reference to a node the walk reaches under a +different one, so an address comparison would answer "not focused" for every element +and the marker would silently never appear. + +**Alternate child collections** (`AXRows`, `AXVisibleChildren`, merged with +`AXChildren`). `AXChildren` alone is not the whole tree: a table, outline or list +routinely exposes its rows ONLY through `AXRows`, and a scrolled list only the +on-screen ones through `AXVisibleChildren`, reporting an empty or scaffolding-only +`AXChildren`. A children-only walk therefore rendered a spreadsheet, a Finder list or +a mail inbox as a container with nothing in it — which reads as "this app has no +content" rather than "you looked through the wrong attribute". For a row-bearing role +the alternates are read FIRST so the rows take the low indices a model actually +addresses. Merged results are deduplicated by element **identity** (`CFEqual`), since +a row is commonly in both collections and each read mints a distinct reference: a +duplicated row is worse than a missing one, because the model would address two +indices believing they are two different rows. `MAX_CHILDREN_PER_NODE` bounds the +**merged** list, so three collections cannot together exceed what one was allowed to. + +**Secure elements disclose only their existence.** No title, no value, no traits and +no frame. `editable` would confirm the password box accepts input and a rect would +locate it precisely enough for a coordinate click, so both are withheld for the same +reason the value is. The selection read is gated on the **focused** element +specifically rather than the window's `has_secure`, because a window may hold a +password field while the caret sits in an ordinary search box — refusing there would +withhold something that is not sensitive. + +### The click ladder's last rung: the enclosing control + +`AXPress` → `AXConfirm` → `AXOpen` recovers most element clicks. What it does not +recover is web content, which renders a clickable row as a plain `AXStaticText` +inside a pressable wrapper: the text node advertises no actions and refuses every +verb, so an element click reported failure for a row a human clicks without +thinking — leaving coordinates as the only move, which is what the element path +exists to avoid. + +So a failed ladder tries the nearest ancestor that plausibly IS the control. An +unguarded version of this is a wrong-click generator (climb far enough and something +is always pressable — eventually the page), so it is bounded twice: + +- **hops** (`MAX_ANCESTOR_PRESS_HOPS = 3`) — beyond that the ancestor is more likely + the page than the row; +- **area** (`MAX_ANCESTOR_AREA_RATIO = 8.0`) — the real signal that an ancestor is + "the same control" is that it is roughly the same SIZE. A row wrapping a text node + is a small multiple of its area; a scroll area or page body is orders of magnitude + larger. **Without the target's own frame there is nothing to compare against, so + the fallback declines rather than guessing** — and declines before spending any AX + round-trips on a climb it could not judge. + +Only `AXPress` is attempted (not the full ladder): `AXOpen` on a container can mean +something quite different from activating the row inside it. It never applies to a +right click, for the same reason the menu ladder never falls back to a press — a +context menu on the wrapper is a different menu. And the result **says** it pressed +the enclosing control, because the model has to be able to tell "my click worked" +from "something near my click worked". + +--- + +## Index lifecycle (and its honest limit) + +Element indices address a LIVE user interface, so the snapshot cache +(`index.py::SnapshotIndex`) is a correctness control, not an optimization. + +**Entries are keyed by `(session_key, window_key)`** — and both halves of that key +came from a review finding. + +*The window half.* Element indices address one WINDOW's accessibility tree, so +keying by application alone aliased distinct windows of the same app: snapshot +document A, focus document B, and the follow-up action — which re-resolves to B — +retrieved A's cached tree. The fingerprint check cannot catch that, because two +documents of the same app routinely have identically-shaped toolbars, so +`role|subrole|title` at a given index matches and the action mutates the wrong +document. `AppRef.window_key` is the app identity plus **pid and window id** (a +window id is only unique within a session, and a relaunched app can reuse one). +`AppRef.key` deliberately stays window-agnostic — it is the denylist and +allow/deny-pattern identity, where "block Terminal" must mean every Terminal window. + +*The session half.* The gateway is +one process serving every surface — dashboard tabs, Slack threads, cron jobs — so an +app-only key made the cache shared mutable state across concurrent sessions: session +A walks Preview and is shown indices, session B then walks Preview after the UI +moved and its snapshot REPLACES A's, and A's next action resolves *and* +fingerprint-verifies against B's tree. Both sessions look internally consistent and +the wrong control is activated. Lifecycle is per-session too: `computer_end_turn` +drops only the calling session's entries (a process-wide clear would let any surface +turn another's next action into a spurious "call `computer_get_state` first"), and +`MAX_INDEXED_APPS` is a per-session cap so a chatty surface cannot evict another's +live indices. `SnapshotIndex.clear()` remains the process-wide reset, reachable only +from lifecycle callers (a backend swap), never from a tool. Namespacing removes the +CROSS-session race entirely; it does not remove the within-session one below, which +is inherent to driving a live UI. + +Three mechanisms bound a single session's validity, none of which needs a +turn-boundary signal kiro-cli does not have: + +1. **Hard fail, never lazy re-snapshot.** Acting on an app with no cached + snapshot is refused: `Error: no state for 'Finder'. Call computer_get_state + first.` A lazy re-walk would let the model act on a tree it was never shown, + which is exactly the failure indices exist to prevent. +2. **TTL** — `SNAPSHOT_TTL_SECS = 90`, `time.monotonic()` throughout so a clock + adjustment cannot make a stale snapshot look fresh. Expired: + `Error: state for 'Finder' is 214s old. Call computer_get_state again.` + `MAX_INDEXED_APPS = 8` bounds RSS (each snapshot holds its encoded JPEG). +3. **Fingerprint drift** — the real guard, run unconditionally before every + mutating action against a FRESH walk (measured 40-70ms: Finder 145 nodes / + 0.04s, Chrome 1431 / 0.07s). On drift: `Error: element_index 7 changed since + the last computer_get_state (was 'AXButton "Save"', now 'AXButton "Delete"'). + Call computer_get_state again.` The fresh walk becomes the new cached + snapshot. The fingerprint is `role|subrole|title` — `value` is deliberately + excluded, because a text field's value changes as the user types without the + control's identity changing, and folding it in would refuse almost every + legitimate action. A secure record contributes only role/subrole/title, so + fingerprinting never reads credential bytes. + +Plus `computer_end_turn` for explicit early release, and `reset_shared_backend()` +drops the cache too (indices from one driver's walk are meaningless against +another's). + +**Honest limit — fingerprinting narrows the race, it does not eliminate it.** A +tree can still change between the verifying walk and the action microseconds +later. It converts a silent wrong-click into a loud refusal in the overwhelming +majority of cases; it is **not** a transactional guarantee, and the spec says so +rather than overclaiming. Every mutator re-snapshots, and every mutator is +interactively approved by default so a human sees the prompt. + +Indices are also *dense*: elidable containers (`AXGroup`, `AXUnknown`, +`AXSplitGroup`) with no title or value are dropped WITHOUT consuming an index, +and `index.resolve()` looks an element up by its own `index` field rather than by +list position — position and index are not interchangeable. + +--- + +## The ctypes layer: four findings from running real code + +All native work is confined to `macos_ffi.py`. `import ctypes` is a top-level +import statement (AUTOSDE `top-level-imports`), but **no `CDLL`/`find_library` +runs at module scope** — the four frameworks (CoreFoundation, +ApplicationServices, CoreGraphics, ImageIO) load inside `_frameworks()`, cached +in a module global, raising `ComputerUseUnsupported` off macOS. A module-level +`CDLL` would raise `OSError` on the Linux CI fleet at import time and break +collection of every test that transitively imports the package. + +**1. A missing `argtypes` is a SIGSEGV, not a TypeError.** ctypes marshals a +Python int as a 32-bit C int and TRUNCATES the 64-bit pointer. This produced a +real `EXIT=139` on the first prototype run; adding explicit +`CFGetTypeID.argtypes` / `CFStringGetLength.argtypes` / `CFRelease.argtypes` +fixed it. Therefore: one declarative `_FN_SPECS` table, one bind pass at first +`_frameworks()` call so no function can be reached un-bound, and `_bind` RAISES +when `argtypes is None`. `CGPoint`/`CGSize`/`CGRect` are real `ctypes.Structure`s +— passing two doubles where a struct is expected mis-marshals the call. A test +asserts every row has both a non-None `argtypes` and a `restype`, and that +`CGEventPost(` appears nowhere in the module while `CGEventPostToPid` does. + +Related hygiene: `_cf_string()` is a context manager that `CFRelease`s on exit (a +tree walk creates thousands of CFStrings; leaking them is a real RSS bug the +watchdog would eventually recycle the session over), and `ax_attr()`/`ax_str()` +type-check `CFGetTypeID(v)` against `CFStringGetTypeID()`/`CFArrayGetTypeID()` +**before** using a value — a wrong-type read is another segfault, not an +exception. + +**2. Electron/Chromium apps need an explicit opt-in.** Chrome returned +`kAXErrorCannotComplete = -25204` for every attribute read. Setting +`AXManualAccessibility = kCFBooleanTrue` on the app element and waiting ~2s +unlocked **1431 nodes in 0.07s**. Without this, Slack, VS Code, Obsidian and +KiroCrew's own desktop app appear permanently empty. Order of operations: +create the app element, immediately +`AXUIElementSetMessagingTimeout(app_elem, AX_MESSAGING_TIMEOUT_SECS)` — +mandatory, because ctypes releases the GIL around the C call and a genuinely hung +target app would otherwise park the worker thread indefinitely — read +`AXWindows`, and on `-25204` set the opt-in, poll for up to +`ELECTRON_OPT_IN_WAIT_SECS` in 0.25s steps, retry **once**, then raise naming the +raw AX code so a support thread is diagnosable. The first `computer_get_state` +on an Electron app is inherently slow; the skill says so, so a model does not +read it as a hang. + +**3. `pgrep` is the wrong way to resolve an app to a pid.** +`pgrep -n "Google Chrome"` returned 47492 — a short-lived helper that vanished +seconds later and answered `-25204` to everything — while the real browser was +637. `pgrep -n Slack` gave 1614 (helper) against the window list's 942 (real). +Resolution is therefore ALWAYS from `CGWindowListCopyWindowInfo` with +`kCGWindowListOptionOnScreenOnly | kCGWindowListExcludeDesktopElements`, keeping +`kCGWindowLayer == 0` entries and taking `kCGWindowOwnerPID`. `list_apps()` is +built from the same window list and nothing else. A test asserts the source text +contains no `pgrep`. + +**4. Synthesized key events inherit the user's LIVE modifier state.** Typing +`"a","b","c"` into TextEdit produced **`' I Abc'`**. Three rules, all +load-bearing and all verified to then produce exactly `'abc'`: + +```python +src = cg.CGEventSourceCreate(kCGEventSourceStatePrivate) # 1: PRIVATE, never None +ev = cg.CGEventCreateKeyboardEvent(src, keycode, is_down) +cg.CGEventSetFlags(ev, flags) # 2: ALWAYS called, even when flags == 0 +cg.CGEventPostToPid(pid, ev) # 3: app-targeted, never the global tap +``` + +`flags` is built from ZERO and OR-ed with only the caller's parsed modifiers. +`CGEventPost` (global tap, hits whatever is frontmost — i.e. whatever the user is +actually doing) appears nowhere. + +Two smaller findings encoded the same way: **the advertised action list lies** — +an `AXScrollArea` advertising `AXScrollDownByPage` returned +`-25205 (kAXErrorActionUnsupported)`, so `scroll` attempts the AX action, checks +the error code, and falls back to `CGEventCreateScrollWheelEvent` posted with +`CGEventPostToPid`; and **the tree walk is ITERATIVE** (an explicit stack), so a +pathological deep tree cannot `RecursionError` inside ctypes. + +**Secure fields are a subrole, not a role.** A real macOS password box reports: + +``` +AXRole = 'AXTextField' <- looks like an ordinary text field +AXSubrole = 'AXSecureTextField' <- the ONLY reliable signal +AXValue = readable +``` + +Checking `AXRole == "AXSecureTextField"` — the intuitive check — **misses every +password field**. The driver sets `secure = (role == SECURE_SUBROLE or subrole == +SECURE_SUBROLE)`, and three protections key off that one flag: `render` emits +`` for the value (and for the title, which is sometimes the account +name), `policy.check_input_target` refuses `set_value`/`type_text`/`press_key` at +a secure target, and a window containing ANY secure node gets **no screenshot at +all**. Whole-window suppression rather than a blanked rectangle: there is no +reliable way to blank a sub-rectangle of an already-encoded JPEG, and a partial +redaction that missed would be worse than none. + +**Every walk cutoff sets a flag, and the capture gate treats "unknown" as +"present".** `MAX_CHILDREN_PER_NODE = 512` bounds one pathological container (a +table with 100k rows) before the global node budget notices — but it used to drop +the tail *silently*, so `saw_secure` reflected only the first 512 children while +`truncated` stayed False. The walk then reported itself complete and non-secure and +`capture_snapshot_image` had nothing to refuse on, leaving a password field as child +513 with the whole window's pixels capturable (reviewer finding). A capped read now +sets `truncated`, which already means "there is more of this tree than you were +shown". Detected as `len(children) >= limit` rather than by re-reading the array for +a real count (that would reintroduce the cost the cap exists to avoid), so a node +with exactly the cap many children is treated as truncated too — a false positive +costs one suppressed screenshot, a false negative is the disclosure. + +**Every refusal is audited, including the pre-gate ones.** The gate audits its own +denials and `_audit_allowed` records permitted calls, which left a hole between +them: a schema `ValidationError`, an unknown tool, a bad `click_method`, a stale +index, an unparseable key and the paste refusal all return through `_refusal` +*without* reaching the gate, so nothing was recorded (reviewer finding). An audit +trail with a gap at "malformed or refused attempts" is the wrong shape for this +surface — a burst of them is exactly the signal an investigation wants. There are two audited exits and no third: `_refusal` +for text that can quote the desktop (it also traverses the observation ceiling and +the redaction pass), and `_static_refusal` for this package's own static prose about +the caller's request — an unregistered tool, the feature being disabled, a missing +`element_index`, a coordinate form under a targets ceiling, a malformed pointer +request, a governance denial. The second helper exists because six of those sites +returned `f"{ERROR_PREFIX}{…}"` inline and so were still unaudited after the first +fix; an AST test now asserts no other function in the module builds that string, so +a seventh cannot be added unaudited. Both emit a `refused` `log_tool_invocation` +with the tool name and **no resources**: the refusal text can quote a window title and this event fires before +the observation ceiling has been applied to it, so the audit line carries the fact, +never the desktop detail ("redacted credentials" is a weaker guarantee than "never +included"). + +**Paste is refused outright.** `computer_press_key` rejects any Command+V or +Control+V chord (`keymap.is_paste_shortcut`, keyed on the RESOLVED keycode+flags so +`command+V` / `super+v` / `meta+v` / `cmd+shift+v` cannot spell around it). The +clipboard is out of band: KiroCrew never reads it, so nothing can classify what it +holds, and a paste into an ordinary readable field puts that content into the tree +the very next snapshot returns. The secure-target refusal cannot help here — the +*destination* is not a secure field, and the credential arrives from outside every +channel that gets inspected — so the disclosure is "whatever the operator last +copied", which is routinely a password from a password manager's copy button. +Typing known text stays available and is the pointer the refusal gives: the content +of `computer_type_text` is inspectable, so the sensitive-text scan can actually run +on it. + +**Keyboard input refuses when the target will not take focus.** Both keyboard +tools require `element_index` precisely so the addressed element can be inspected +for that flag — and the macOS driver sets `AXFocused` on it before typing so the +keystrokes land there. That focus attempt used to be best-effort: on failure it +logged at debug and typed anyway, which delivered the input to whatever the app +focused LAST — an element no check ever saw, and possibly the password box the +refusal above exists to protect (reviewer finding). `type_text` and `press_key` now +return a failed `DriverResult` naming what did *not* happen ("the keystrokes were +NOT sent"), because a model told only "focus failed" would assume the input landed +and go verify a change that never occurred. + +### Threading + +The gateway is async and the native work is blocking, so every accessibility +walk and capture is offloaded: +`await loop.run_in_executor(subprocess_executor(), fn, ...)`. + +`subprocess_executor()` and never the DEFAULT pool — for both `tools.dispatch` and +the gateway handler's `_dispatch_off_loop`. That pool is the one the repo reserves +for calls that can block on a wedged external resource, and a hung target app +parking a worker for the whole `AX_MESSAGING_TIMEOUT_SECS` is exactly that: on the +default pool a few wedged desktop calls would starve every other +`run_in_executor(None, …)` user in the gateway and the loop's own `getaddrinfo`. +(The handler's *config* reads/writes do use the default pool — short, bounded +filesystem work.) Verified safe: +4 concurrent AX walks in threads returned identical node counts (146 — no CF +thread-safety corruption) while an asyncio heartbeat logged **40/40** ticks, so +the loop never stalled. The AX messaging timeout above is what bounds the +worst case of a hung target app parking a worker. + +--- + +## Screenshot pipeline (measured, not guessed) + +Capture and encode are **100% in-process ctypes** — +`CGWindowListCreateImage` → ImageIO `CGImageDestinationCreateWithData` / +`AddImage` / `Finalize` with exactly two option keys +(`kCGImageDestinationImageMaxPixelSize`, `kCGImageDestinationLossyCompressionQuality`). +No subprocess anywhere in the package, and therefore no Pillow dependency +(Pillow is declared in neither `setup.cfg` nor `pyproject.toml`, and a subprocess +node would need a `test_spawn_audit.py::BENIGN_SPAWNS` entry). All of +`img`/`data`/`dest`/`opts` and every CFNumber/CFString are released in a +`finally`, including when `Finalize` returns `False` (which degrades to a +tree-only result rather than raising). + +Compression defaults are computer use's **own** 1280px / q55 — deliberately NOT +browse's 1920/q70 — because the accessibility tree is the primary channel and +the image is corroboration. Measured on a real 1840x872 window: + +| Width | Quality | Bytes | ~Tokens | Note | +|---:|---:|---:|---:|---| +| 1840 (orig PNG) | — | 123,343 | 41,115 | unusable | +| 1920 | 70 | 48,889 | 16,297 | browse's defaults — only 60% off | +| **1280** | **55** | **24,766** | **8,256** | **shipped default**, visually verified fully legible | +| 1024 | 55 | 17,918 | 5,973 | still readable, tighter | +| 800 | 40 | 10,781 | 3,594 | small text starts to soften | + +The 1280/q55 output was decoded and inspected: every sidebar label, filename, +toolbar icon and the selected-item highlight are clearly readable. For contrast, +the reference runtime attached a native-resolution PNG unconditionally and no +parameter capped it — a Slack window measured **~437KB base64 ≈ 109K tokens** on +one call. + +Files land in `os.path.join(tempfile.gettempdir(), "kirocrew-computer-shots")` +(the exact idiom `mcp_playwright_proxy.py` uses and which its test pins by source +text), created `mode=0o700`, each file passed through +`platform_compat.restrict_to_owner`, ring-trimmed to `SCREENSHOT_KEEP = 200`. +Only the **path** is ever relayed; the bytes never enter a result. + +--- + +## The live view (PiP) + +`computer_use/screencast.py` + `website/src/components/ComputerUseLiveView.tsx`. +The machine being driven is often not the machine the operator is looking at (a +cloud Mac, a session reached over the reverse SSH tunnel, or another Space), so a +floating picture-in-picture panel mirrors what the agent sees. Same shape as the +browse mirror (`browser/screencast.py` → `/api/browser/frame` → WS → +`BrowserLiveView`), for the same reason: it rides an existing capture rather than +opening a new one. + +**It is a RELAY, not a capture.** `capture_snapshot_image` hands the JPEG it just +encoded for the model to `emit_snapshot_frame`. There is no timer, no second +`CGWindowListCreateImage`, no full-screen grab, and no way for the panel to ask +for a frame — so opening it cannot make the agent screenshot anything, and it can +never show a window the model did not already read. Frames are therefore sparse +(one per `computer_get_state` with `attach_screenshot`) and are always the +already-downscaled 1280px/q55 bytes. + +**Three suppressions, all evaluated before anything leaves the process** (the +ingress handler is NOT the boundary and is not relied on as one): + +1. **No published surface scope → no frame.** The capture layer has no session + identity of its own (`SnapshotRequest` carries budgets), so + `api_computer_use_invoke` wraps its one blocking dispatch in + `screencast.frame_scope(...)`, publishing `(session_key, agent, app)` on the + worker thread. It is `threading.local` — not a contextvar, which would not + survive the executor hop, and not a module global, which would leak one + surface's identity into the next dispatch on the same pooled thread — and it + restores the previous value on exit. A capture reached any other way (a CLI + probe, a future caller that skipped the handler) emits nothing rather than + guessing an identity, matching the gate's fail-closed treatment of an empty key. +2. **A secure window is never mirrored**, read from `Snapshot.has_secure` — the + driver's own predicate, the one `capture_snapshot_image` already refuses on — so + there is exactly one definition of "this window holds a password field". +3. **A withheld `screenshot` channel emits nothing**, via + `gate.permitted_observation_channels`: the same evaluator the tool path and the + Settings snapshot use. That evaluator permits every channel today, so this is + the seam that would hold if an observation ceiling were reintroduced, not a live + restriction. + +**Wire.** `POST /api/computer-use/frame` (loopback + `X-Internal-Secret`, strict, +both re-asserted in the handler) carries +`{data, format:"jpeg", width, height, session_key, app}`. +`build_frame_payload` bounds every field: base64 charset (which structurally +excludes `:`, whitespace and `<`/`>`, so no URL and no markup) plus a +`MAX_FRAME_B64_CHARS` cap, `format` restricted to the single value `"jpeg"` (a +frame claiming PNG or WebP is **refused, not relabelled** — that is what makes +"never a full-resolution PNG" structural), the dimensions to positive ints within +`MAX_SCREENSHOT_MAX_PX` with `bool` excluded, and the two text fields to explicit +charsets. `app` is the resolved application's display NAME, never the window title +(titles are their own observation channel and can carry document names and paths). + +The rebroadcast is `deliver_ws_owners`, **not** `broadcast_ws`: an App Kit +credential can open `/api/ws` and lands in the all-clients set, and a live view of +the operator's desktop must not cross that boundary. One SEL line per frame +records only the delivery count — never the pixels or the mirrored app, so the +audit log does not itself become a record of what was on screen. + +The POST runs on a daemon thread (the capture runs in a worker thread, and +`broadcast`/`deliver` are event-loop objects — `ensure_future` off-loop raises) and +swallows every failure; `capture_snapshot_image` additionally wraps the call so +its own "never raises" contract cannot be broken by a decorative mirror. + +**Panel.** `hidden → (first frame) open ⇄ chip`, docked bottom-LEFT so it never +stacks with the browse mirror's bottom-right. Draggable, eight resize grips, two +size presets, size persisted in `localStorage`, geometry always fitted back into +the viewport. Close remembers the dismissed session so its later frames do not +re-open the panel; a different driving session still surfaces. Read-only — no +click-through, no input relay, no control channel. The empty state states both +reasons a frame may be absent, so "nothing here" never reads as a fault. + +--- + +## Cursor Motion: a real desktop overlay, cosmetic only + +`computer_use/cursor_motion.py` (geometry) + `overlay.py` (gateway supervisor) + +`overlay_proc.py` (the AppKit child). A **fake cursor drawn on the operator's real +desktop**, animated along the path a click is about to take. It exists for one +reason: an unattended desktop action is otherwise unreviewable in real time — a +human glancing at the screen sees a window change with no indication of what +caused it. The overlay makes "the agent is about to press *that*" visible. + +It is **purely cosmetic and never on the success path of a tool.** Nothing it does +can make a call fail, and nothing it draws can make a call succeed. Three +structural properties enforce that: + +- **It is not the pointer.** The overlay is a drawn image; the physical cursor is + untouched. Cursor Motion and `click_method: "global"` are independent and + unrelated — the overlay can decorate an `app_post` click that moves nothing, and + the pointer path works with the overlay off. Turning Cursor Motion on grants no + new capability, which is why it is an ordinary `config.json` preference + (the typed `ComputerUseConfig.cursor_motion` field, **default OFF**) rather than + a keystone flag. + +**Where it is invoked.** `overlay.show_pointer_motion(x, y, count)` is the sync +seam the blocking dispatcher calls from `tools._perform`, immediately before a +gesture whose `moves_pointer` is true — i.e. only `click_method: "global"` clicks +and drags. The app-scoped and accessibility paths never move the physical cursor, +so animating one there would show a gesture that is not happening. The dispatcher +runs on a worker thread, so the seam schedules the animation onto the gateway loop +with `run_coroutine_threadsafe` (bound per request by the invoke handler, which +runs *on* that loop) and does **not** await it: waiting would add the glide's +duration to every pointer click. Ordering is therefore best-effort — the click may +land a few milliseconds before the drawn cursor finishes arriving — which is the +deliberate trade against putting an animation on the critical path. + +The Settings row appears only on macOS, and only once computer use itself is +enabled — a glide with no pointer click to accompany it would advertise something +that never happens. +- **It never raises and never blocks the loop.** Every `CursorOverlay` method + swallows every exception (a dead child, an unavailable AppKit, a full pipe all + degrade to "no visual cursor") and the animation is fire-and-forget: the command + is written and the coroutine returns rather than awaiting ~1.4s of decoration on + the latency path of a real action. Spawn is `asyncio.create_subprocess_exec` with + one bounded `wait_for` on the readiness line. +- **It is screenshot-invisible BY DESIGN.** The child sets + `NSWindowSharingNone` (`setSharingType: 0`), verified A/B: the overlay renders on + the desktop and does not appear in `screencapture` output. That is a correctness + requirement, not a nicety — the agent screenshots the same screen it is drawing + on, and a fake cursor in the pixels the model reads would be an artifact the model + reasons about as if it were part of the application. + +### Why a separate process + +AppKit needs a main-thread run loop and the gateway's main thread **is** the +asyncio loop, so the overlay cannot live in the gateway at all. `overlay_proc.py` +is therefore its own process (`python -m kiro_crew.computer_use.overlay_proc`, +fixed argv, nothing agent-supplied in it) driven by newline-delimited JSON on +stdin. It has its **own ctypes surface**, which is the one documented exception to +"`macos_ffi.py` is the only module that touches ctypes": that invariant exists so a +native fault cannot take down the gateway, and a fault in a separate short-lived +child costs one animation. The child's docstring says so. + +`overlay.py` serializes on one `asyncio.Lock` — the child's stdin is an ordered +byte stream and two concurrent writers would interleave half-lines — and the lock +is created lazily, because an `asyncio.Lock` built outside a running loop is the +cross-loop hazard `kiro_crew/__init__.py` documents. + +Off macOS, and whenever the opt-in is off, `cursor_motion_enabled()` returns False +before anything else in every method, so a Linux CI shard exercises these bodies +and observes that no child is spawned. `show_pointer_motion` additionally no-ops +when no gateway loop is bound — a coroutine scheduled onto a loop nobody runs would +never execute, and a closed loop (a gateway restart) is treated the same way. The +config read stays `getattr`-based even though the field is now declared, and fails +to OFF: an unreadable setting can only ever mean "no decoration", never "start +drawing on the user's screen". + +### The path model is pure geometry + +`cursor_motion.py` has **no ctypes, no subprocess, no AppKit, no config read** — +given a start and an end point it returns sampled screen points plus a duration, +and every function is total in its arguments. That is what makes the *feel* of the +animation unit-testable on a display-less CI shard: a regression in how the cursor +moves fails an assertion on numbers rather than requiring somebody to watch a +screen. The shape is one cubic Bezier bowed by a perpendicular arc +(`clamp(distance * 0.22, 28, 110) * curve_scale`), asymmetric control points so the +cursor leaves fast and arrives settling, driven by a velocity-Verlet spring on +**progress** (not position) at a fixed 1/240s step — so the spring's slight +numerical overshoot past 1.0 is clamped before sampling and the drawn cursor can +never overshoot the point it is advertising. + +Coordinates are **top-left** everywhere in this module, matching the AX/CG +surfaces and `screencapture -R`. The bottom-left flip AppKit's `NSWindow` origin +needs (`y_bottom = H - y - h`) happens in `overlay_proc` alone, at the one place +that actually talks to AppKit. + +--- + +## Always-on floors (no policy key, and none will be added) + +These are the protections that survive the governance removal, and they survive +BECAUSE they were never governance keys. Each is unconditional code — no policy +file, no profile, no toggle — which is exactly why removing the ceiling did not +touch them. They sit with `_SENSITIVE_HOME_DIRS` and the AKIA redaction: + +- **Secure-field value redaction** — `render` emits ``; the value bytes + are never rendered, not truncated, not masked-with-a-hint. +- **Whole-window screenshot suppression** when any node is secure. +- **Credential / exfil redaction** — every renderer ENDS with + `policy.redact_result`, which routes through `platform.redact_via_context` (so + a loaded companion's extra credential and cookie patterns apply). This is the + primary egress control for tree text, not belt-and-suspenders: live probes + observed real filesystem paths, mounted volume names, bundle ids and document + names in accessibility trees and window titles. +- **Refusals go through the SAME exit as results** (`tools._refusal`). A refusal is + prose about the desktop: a fingerprint-drift message embeds + `render.describe_record` for the cached AND the fresh element (two verbatim + accessibility titles), a driver failure quotes the app label, `check_app` names + the resolved bundle id. So `_refusal` applies both controls in the same order the + result path does — `gate.apply_observation_ceiling` over the sentence as the + `text` channel, then `policy.redact_result` — and when `element_values` is denied + it replaces every quoted fragment with ``, keeping only the + actionable "call `computer_get_state` again" half. Without this, provoking a drift + would be the one path around both the redaction pass and the observation ceiling. + Refusals that are 100% KiroCrew's own static prose (the primary-enable refusal, the + generic governance denials, the "pass an `element_index`" hint) skip it by + construction: no desktop text to leak, and redaction could only mangle them. +- **Per-call SEL audit** — every permitted call emits `log_tool_invocation`, every + denial `log_governance_decision`. The redaction inside those writers is + load-bearing: a denied `set_value` reason can carry the text the agent tried to + type, and an `item` can be a path-bearing window title. + +**Refuse any future request for a policy key that re-enables secure-field +values.** + +--- + +## Platform support + +| Platform | State | +|---|---| +| macOS | Supported. ApplicationServices AX + CoreGraphics + ImageIO via ctypes. | +| Windows | `WindowsBackend(UnsupportedBackend)` — typed refusal. The plan is UIAutomationCore + `PrintWindow`; not implemented. | +| Linux | `LinuxBackend(UnsupportedBackend)` — typed refusal. The plan is AT-SPI over D-Bus; Wayland has no unprivileged window capture (it needs xdg-desktop-portal), so a first cut may be tree-only, which `SnapshotRequest.want_image` already accommodates. | + +Both non-macOS backends **refuse rather than raise**, mirroring how +`dashboard/handlers/terminal.py` handles the Windows PTY, and both name what is +missing so a user learns something and a maintainer finds the next step. When +`supported` is false the Settings panel renders the reason and no toggle. + +`kirocrew computer doctor --json` is what the gateway shells for the Settings +permission rows — a short-lived subprocess, deliberately NOT an in-gateway ctypes +call, so a native fault cannot take the gateway (and with it cron, Slack and the +dashboard WS) down. + +### Permission probes are ADVISORY, never a gate + +`AXIsProcessTrusted()` + `CGPreflightScreenCaptureAccess()` only. Never +`CGRequestScreenCaptureAccess` — it pops a system dialog from a background +process. macOS attributes a TCC grant to the **responsible parent** of the +process tree, so both probes read `missing` while a full-fidelity capture +succeeds; that was observed live. The probe returns a `responsible_hint` naming +the process a user should actually grant (the packaged app, or the terminal that +launched a dev gateway), the Settings copy says "Not detected does not always +mean unavailable", and the feature is **never** gated on the result. Ad-hoc +re-signing anything in the chain can void a grant — the same mechanism that +permanently broke the reference bundle's Accessibility 3/3 times — so +`packaging/resign-macos-libs.sh` is a known hazard for this feature. + +--- + +## The CLI (`kirocrew computer`) — and what is deliberately not ported + +`computer_use/cli.py`. Three verbs, hand-rolled dispatch mirroring +`browser/cli.py`. Full command reference in [cli.md](cli.md). + +| Verb | For | Gated on the primary enable? | +|---|---|---| +| `doctor [--json]` | platform support + enable state + the advisory TCC probe | no — it *reports* the enable, and returns no desktop content | +| `apps` | the on-screen application list | **yes** — see below | +| `call [k=v …]` / `call --calls '[…]'` | run one tool, or a sequence in ONE process | **yes**, and everything else too | + +`apps` is gated, and used not to be. The original reasoning — "this is the operator +running a diagnostic in their own terminal, and the enable exists to stop the +*agent*" — does not hold, because **the agent can run this command with bash**. As +written it was an ungated read of every window TITLE (document names, paths, and +whatever a terminal put in its title) that worked with the feature disabled, in an +unattended cron session, and under a policy banning computer use outright. It now +runs `computer_list_apps` through `dispatch_tool` like everything else, so the app +denylist and the observation ceiling apply. `doctor` remains the ungated way to find +out *why* the feature is off: it reads the keystone and the TCC state only, never the +window list. + +### `call` is a harness, not an eleventh tool + +`call` runs the existing tools through `tools.dispatch_tool` — **the same ordered +chokepoint an agent call traverses**. The primary enable, the fail-closed +`gate.require_computer_use`, the app denylist, index freshness, the secure-target +refusals and the observation ceiling all apply, so `call` cannot see or do anything +the agent could not. That is precisely what makes it a faithful reproduction tool +rather than a debug backdoor, and it is why the implementation goes through +`tools` rather than reaching into `service` (a test asserts that over the AST — a +future "skip the overhead" edit would otherwise stay green while dropping +governance on the floor). + +One consequence, stated rather than left to be discovered: the session key is +always the attended `cli_chat` surface (`sel._infer_source` → `cli`), used for the +SEL audit record. There is no identity proof and no separate diagnostics opt-in — +both existed only to satisfy the unattended-surface refusal, which is gone. `doctor` +remains the diagnostic that works regardless of the enable, because it reads the +keystone and the TCC state and returns no desktop content. + +**Why a whole array in one process.** `element_index` values only mean anything +relative to the `computer_get_state` that produced them, and that mapping lives in +a per-process `SnapshotIndex` with a 90s TTL. Two separate `kirocrew computer call` +invocations therefore cannot share indices at all — the second refuses with "no +state for …". `--calls '[{"tool": …, "args": {…}}, …]'` exists so a +snapshot-then-act sequence is reproducible from one command line. It runs +sequentially and does **not** abort at the first error: a reproduction is more +useful whole, since "step 2 was refused and step 3 then hit a stale index" is the +actual story. `--json` emits `[{tool, text}, …]`; the exit code is non-zero if any +reply carries the `Error: ` prefix. + +`call` has **no MCP twin**, and that is the MCP-first rule being followed rather +than bent: the rule exists so the model gets a structured tool instead of being +told to shell out, and the model already has all ten tools. A tool that runs other +tools would let a model launder one per-call gate decision into many. + +### Deliberately not ported from the reference implementation + +| Reference surface | Why not | +|---|---| +| ~~`sky_click` (a fifth `click_method`)~~ | **Now ported** — see "`sky_click` — the private path, and why it IS shipped" above. Kept in this table as a pointer, because the reasoning that once excluded it (do not depend on private ABI) still governs how it is contained: quarantined in `macos_skylight.py`, never reachable from `auto`, and fully degrading when a symbol is missing. | +| `install-codex-mcp`, `install-claude-mcp`, `install-gemini-mcp`, `install-opencode-mcp`, `install-codex-plugin` | N/A by design. KiroCrew **self-registers**: `kirocrew-computer` is a managed server in `agent.py:_MANAGED_MCP_SERVERS`, auto-written into the agent config and refreshed while preserving user customizations. There is no external host to install into, so an install verb would have nothing to do. | +| `snapshot ` | covered by `apps` + `computer_get_state`, and a CLI spelling of an LLM-facing capability is exactly what the MCP-first rule asks us not to add. | +| `turn-ended [--previous-notify]` (a host notify hook) | same intent, different shape: `computer_end_turn` is an MCP tool, so the model drops its own snapshot cache rather than relying on a host lifecycle hook KiroCrew does not have. | + +--- + +## Known limitations + +Stated plainly. Each one is real; none is papered over. + +### The keystone is the whole security boundary + +There is no second plane any more. `hooks.on_tool_call` still classifies a +computer-use title for the ordinary tool/mcp scopes, but it makes no computer-use +decision, and `gate.require_computer_use` only audits. Everything rests on one +fact: **the keystone `computer_use.json` is on `security._SENSITIVE_HOME_DIRS`, so +the agent can neither read nor write it.** + +That is a real boundary — it is the same mechanism that protects +`security_policy.json` and the SEL HMAC key, and it is what makes "the agent cannot +turn on its own desktop automation" true rather than aspirational. But it is the +ONLY one, and the consequences should be stated rather than discovered: + +* once the operator enables the feature, prompt injection that reaches the agent + reaches the desktop. There is no per-app, per-action or per-surface ceiling left + to contain it; +* the one structural defence is that the agent cannot drive KiroCrew's own window + (`policy.check_app`), so it cannot click the enable itself. If that entry is ever + removed, the keystone stops meaning anything. It matches THREE signals, and the + third is load-bearing: the dashboard is also reachable as a **browser tab**, where + the app identity is Chrome's or Safari's and a bundle/name rule cannot fire at all + (reviewer finding). `DeniedApp.title_substrings` matches the resolved window title, + as a substring — the tab title takes a badge prefix (`(3) Kiro Crew`) and popouts a + `… — Kiro Crew` suffix. And because input is delivered per-PID + (`CGEventPostToPid`), `apps_macos.list_apps` prefers a DENIED title over an + innocuous one when one process owns several windows: a dashboard in a background + tab must refuse the whole browser, not just that window. + + Accepted false positive, stated rather than discovered: any window whose title + merely contains "kiro crew" is refused — an editor with this repo open, a browser + reading this PR. The failure directions are not symmetric. Refusing an unrelated + window is visible and recoverable (the refusal names the target); failing to refuse + our own dashboard silently hands the agent its own security settings. A title is + also the only signal available: the driver reads no page URL, and `AXURL` is not + bound anywhere in the package; +* an agent that can write `~/.kiro/agents/*.json` can stop kiro-cli sending + permission requests, so the interactive prompt is not a guarantee either. That is + a pre-existing gap affecting every capability, hardening deferred to its own PR; +* **typing into a terminal window reaches a shell without passing the command deny + floor.** The 137 `BUILTIN_DENIED_RULES` are matched against a `bash` tool call's + command string; a `computer_type_text` into Terminal.app is not one, so + `security.py` sees none of it. The terminal denylist entry that used to cover this + was removed with the rest of the per-app model (it was incomplete by construction — + an IDE's embedded terminal never matched it — and it blocked legitimate use on the + operator's own machine). The sensitive-text scan still inspects what is typed, but + it is a credential/secret filter, not the command floor. Stated here because it is + the sharpest edge of the one-opt-in posture. + +This is a single-user-machine posture: the operator is trusted with their own +desktop, and the product optimises for the feature being usable rather than for +containing a compromised agent. A deployment that needs the latter should not enable +computer use. + +### Other accepted residuals + +- **The screenshot directory stays agent-readable.** Persisted JPEGs live in a + `0o700` temp dir the agent can reach with `fs_read` — the same posture browse + already ships. Computer use widens WHAT can be in frame (any window, not one + browser tab). Mitigations: per-window capture only (never full-screen), + whole-window suppression when any node is secure, ring-trim to 200, and the + existing `cleanup-temp-screenshots.yml`. This design does not widen the posture + and does not claim to close it. +- **"No screenshots" is not "no disclosure."** The accessibility tree itself + leaked real paths, window titles and bundle ids in live probes, and a document + path inside an `AXTitle` is not a credential so redaction will not catch it. + `window_titles`, `file_paths` and `element_values` are separately governable + channels, and the `file_paths` scrub is a pattern pass over the fields we walk, + not a proof of absence — a relative path, or a path split across two AX + attributes, survives it. A fleet that needs a real bound must also narrow + `computer_use.apps`. +- **Governance is not an undo.** Every mutating tool is irreversible in the real + world: a click that sends an email, a `set_value` that changes production + config in an authenticated tab. `effective = POLICY ∩ PROFILE` decides + *whether*, never *how badly*. A fleet that sets only `enabled: true` with no + sibling rows has granted **unbounded** desktop automation, and the Settings + panel says exactly that in its `unrestricted` state. +- **Element-index addressing is inherently racy** — see the honest limit under + [Index lifecycle](#index-lifecycle-and-its-honest-limit). +- **A ctypes fault ends computer use for the rest of the kiro-cli session.** + kiro-cli caches `tools/list` once per session. Mitigation is prevention (the + argtypes tripwire test), not recovery; a per-call fork was considered and + rejected as disproportionate. + +### Enabling restarts the chat sessions (on purpose) + +That same `tools/list` cache is why `PUT /api/computer-use/config` calls +`_reset_all_sessions` whenever the enable **flips**. ACP has no +`tools/list_changed` notification, so a session that started while the feature was +off keeps an empty computer-use tool set for its whole life: the operator enables +it, asks the agent to look at a window, and is told there are no tools. Restarting +is the same remedy `POST /api/mcp/sync` already applies when MCP routing changes, +and for the same reason. + +Deliberately narrow, so a restart is never gratuitous: + +- only on the `enabled` key, and only when the value actually CHANGED — a no-op + re-save must not tear down the operator's session; +- never for the budget knobs (`max_tree_nodes`, `screenshot_max_px`, …), which are + read per call; +- a restart failure never fails the SAVE. The write already landed and was audited; + reporting failure would be a lie, and the fallback is simply the old behaviour + (the new tool surface appears on the next cold session). + +The response carries `sessions_reset` so the panel can EXPLAIN the restart — an +unexplained session reset reads as a crash. Pinned by +`test_computer_use_api.py::TestEnableRestartsSessions`. +- **The live mirror is SPARSE, not a video feed.** [The live view (PiP)](#the-live-view-pip) + is a relay over the screenshots the model already read, so the panel updates once + per `computer_get_state` with `attach_screenshot` and shows nothing at all during a + run of pure actions. That is deliberate — the alternative is a second capture the + agent did not ask for — but it means the panel is not a substitute for watching the + screen, and the Settings copy must not imply otherwise. +- **Coordinate clicking has no pixel→element verification.** Unlike element + addressing — where the fingerprint check turns an index into a real assertion — + a coordinate names a point the OS delivers to whatever is there *now*. There is + no drift check possible for it, so a coordinate click can land on a control that + moved after the screenshot. Mitigation is the model's own: prefer + `element_index`, and re-`computer_get_state` after anything that reflows a + window. +- **`click_method: "global"` inverts the "the pointer never moves" property.** The + model has to name it (`auto` never resolves onto it) and every use is audited under + its own `tool_kind`, but when it does, the agent can aim the operator's physical + cursor at anything on screen — including UI the app-scoped path deliberately cannot + reach. That reach is the point of the path; there is no longer a separate opt-in + gating it. +- **A drag cannot be verified either, and it is coordinate-only.** No accessibility + action expresses a sweep between two points, so `click_method: "accessibility"` is + refused for it rather than approximated. +- **`_CU_ACTION_CLASSES` must stay in sync with the tool list.** A tool added + without a table row is classified `("mutate",)` — fail-closed in both + directions (it can never satisfy an `@observe` allow-list and IS caught by an + `@mutate` deny), but it also means a *read* tool added without a row will + needlessly prompt. The coverage tests enumerate the registered tool set; the + next author adds the row. +- **The shell plane is a separate plane.** `osascript` / `cliclick` / `xdotool` / + `screencapture` typed into a Bash tool are `commands`-scope items, never + re-parsed into GUI sub-effects, and the **web terminal PTY** + (`dashboard/handlers/terminal.py`) contains no deny-floor or governance call at + all — it is an operator-only, ungoverned plane today. Playwright's + `browser_take_screenshot` is a second pixel channel needing its own `mcp` deny. + None of these are covered by any `computer_use.*` scope, and the spec says so + rather than implying coverage. + +--- + +## Files + +| File | Purpose | +|---|---| +| `computer_use/types.py` | Every constant + frozen dataclass; dependency-free and platform-free | +| `computer_use/keymap.py` | Carbon keycodes, CG flag masks, `parse_key()` | +| `computer_use/policy.py` | The one retained app refusal (KiroCrew's own window) + the operator's allow/deny lists, secure-target + text refusals, the click-target/method/button refusals + `resolve_click_method`, `redact_result` | +| `computer_use/render.py` | Tree/app-list rendering, `fingerprint`, secure placeholder | +| `computer_use/index.py` | `SnapshotIndex`: TTL, cap, `resolve`, `end_turn`, drift message | +| `computer_use/enable_state.py` | The keystone primary enable + the operator's app allow/deny lists (read fail-soft to off) | +| `computer_use/backend.py` | `ComputerUseBackend` ABC, `UnsupportedBackend`, registry, the one platform branch | +| `computer_use/gate.py` | The SEL audit of every call and every real-pointer gesture, plus the pass-through shims (`apply_observation_ceiling`, `permitted_observation_channels`) the renderers still route through | +| `computer_use/service.py` | The single dispatch chokepoint (`act()`), synchronous | +| `computer_use/windows_driver.py`, `linux_driver.py` | Typed refusals + the implementation plan | +| `computer_use/macos_ffi.py` | The ONLY module touching ctypes: `_FN_SPECS`, structs, binder, CF hygiene, key/scroll/mouse event synthesis | +| `computer_use/apps_macos.py` | Window-list app enumeration + pid resolution (never `pgrep`). Bundle `Info.plist` reads honour `security.is_sensitive_path`, so a bundle planted under a protected directory resolves to "identity unknown" rather than being opened | +| `computer_use/snapshot_macos.py` | Iterative AX walk, `AXManualAccessibility` retry, secure detection | +| `computer_use/capture_macos.py` | In-process capture + ImageIO encode + `0o700` persistence + ring trim | +| `computer_use/screencast.py` | Live-view (PiP) relay: `frame_scope`, the three suppressions, `build_frame_payload`, the loopback POST | +| `computer_use/cursor_motion.py` | Cursor Motion PATH MODEL — pure geometry (Bezier + arc + progress spring). No ctypes, no AppKit, no config | +| `computer_use/overlay.py` | Gateway-side overlay SUPERVISOR: the `cursor_motion` opt-in, child lifecycle, motion commands. Never raises, never blocks the loop | +| `computer_use/overlay_proc.py` | The AppKit overlay CHILD (`python -m …overlay_proc`). Its OWN ctypes surface — out of process on purpose; `NSWindowSharingNone` keeps it out of screenshots | +| `computer_use/permissions.py` | Advisory TCC probe + `responsible_hint` | +| `computer_use/macos_driver.py` | `MacOSBackend` glue | +| `computer_use/cli.py` | `kirocrew computer doctor [--json] \| apps \| call` | +| `mcp_computer.py` | The thin stdio shim (`kirocrew mcp-computer`) | +| `testing/fake_computer_use.py` | `FakeComputerUseBackend`, shipped in the wheel | +| `dashboard/handlers/computer_use.py` | `/api/computer-use/{config,invoke,frame}` | +| `website/src/pages/settings/ComputerUsePanel.tsx` | Settings → Computer Use | +| `website/src/components/ComputerUseLiveView.tsx` | The floating live view (PiP) panel | +| `website/src/hooks/useComputerUseFrame.ts` | Frame-stream subscription + session-title lookup | +| `src/kiro_crew/builtin_skills/computer-use/SKILL.md` | The agent-facing workflow. **Bundled**, not in the top-level `skills/` dir: `config/prompt.md` tells the model to read it by name, so per AGENTS.md it is load-bearing and must reach every pip/DMG install | + +Cross-references: [governance.md](governance.md) for why computer use is +deliberately NOT governed; [security.md](security.md) for the keystone leaf and the +denylist's place in the security model; [config.md](config.md) for the +`computer_use` config section; [cli.md](cli.md) for the commands. diff --git a/docs/system-specs/modules/config.md b/docs/system-specs/modules/config.md index 4f206666ff5..4788b5e7a1b 100644 --- a/docs/system-specs/modules/config.md +++ b/docs/system-specs/modules/config.md @@ -390,6 +390,17 @@ class SttConfig: device: str = "cpu" # "cpu" or "cuda" timeout_secs: int = 300 +@dataclass +class ComputerUseConfig: + # DISPLAY + LIMITS ONLY. There is deliberately NO `enabled` field — see the + # note under "Computer use: no enabled field here" below. + max_tree_nodes: int = 1200 # accessibility-tree node budget per snapshot + max_tree_depth: int = 64 # depth budget (the walk is iterative, so this is a cost bound) + text_limit: int = 500 # per-element text truncation (chars) + attach_screenshot: bool = True # default for the `screenshot` tool param + screenshot_max_px: int = 1280 # longest-edge downscale (NOT browse's 1920 — the tree is the primary channel) + screenshot_jpeg_quality: int = 55 # JPEG quality (NOT browse's 70); 1280/q55 measured at ~8.3K tokens vs 41K for a raw PNG + @dataclass class MessagingConfig: use_transport: bool = True # route inbound Slack through SlackTransport → TurnDriver → SlackRenderer (the canonical path); false falls back to the native handle_message monolith @@ -445,6 +456,7 @@ class KiroCrewConfig: memory: MemoryConfig knowledge: KnowledgeConfig stt: SttConfig + computer_use: ComputerUseConfig hooks_data: dict # raw hooks from config.json dashboard_url: str = "" # e.g. "http://my-host.example.com:8080" auto_update: bool = True @@ -453,6 +465,69 @@ class KiroCrewConfig: slack_dm_activation: str = "always" # activation mode for DMs (D-prefix channels) ``` +### Computer use: no `enabled` field here, and no pointer flag either + +`ComputerUseConfig` carries display and limits only. **Two** switches for native +desktop GUI automation live **outside `config.json`**, on the keystone at +`~/.kiro/crew/computer_use.json` (path via `config.loader.computer_use_state_path()`, +leaf on `security._CREW_SECRET_LEAVES`): + +```json +{ + "enabled": false, + "allow_pointer_move": false, + "allowed_apps": [], + "extra_denied_apps": [] +} +``` + +The absence is deliberate and the precedent is `denied_commands.json`: +`is_sensitive_write_path("~/.kiro/crew/config.json")` is `True` (the *tool* path is +protected), but `is_sensitive_bash_command("echo x > ~/.kiro/crew/config.json")` is +`None` — `_WRITE_PROTECTED_BASH_LEAVES` is `('.data-home-ready',)` only. A config +toggle would therefore be flippable by a prompt-injected agent through any shell +redirect. + +- **`enabled`** — the primary enable for full desktop observation plus input + synthesis. A security ceiling, so it goes where the agent can neither read nor + write it. +- **`allow_pointer_move`** — the operator's consent for `click_method: "global"`, + the one path that warps the **real** mouse pointer. Same class of control, same + treatment, for exactly the same reason. It is only half the permit: the + `capabilities.computer_use_pointer` governance row is required in addition + (tightest-wins), and neither substitutes for the other. + +Both are read with a strict `is True` identity test, so a truthy string such as +`"allow_pointer_move": "false"` does **not** hand over the mouse, and both reads +fail soft to `{}` → **off**. See [security.md](security.md), +[governance.md](governance.md) and [computer-use.md](computer-use.md). + +#### `computer_use.cursor_motion` — the one new `config.json` flag + +Cursor Motion (the cosmetic fake-cursor desktop overlay) is the exception that +proves the rule above: it belongs in `config.json` precisely *because* it grants no +capability. `computer_use.cursor_motion` is a **display preference, default OFF** — +the overlay draws an image, never moves the pointer, cannot deliver input, and is +invisible to `screencapture`, so an agent flipping it could at most decorate its own +clicks. A keystone flag would imply a security decision that does not exist. + +`overlay.cursor_motion_enabled()` reads it through `getattr(section, +"cursor_motion", False)` rather than as a typed attribute, which makes the read +**forward-compatible and fail-OFF**: a build whose `ComputerUseConfig` predates the +field resolves to OFF rather than raising inside a tool call, and a missing field can +only ever mean "no decoration", never "start drawing on the user's screen". The +typed `ComputerUseConfig` field (and its `_EDITABLE_CONFIG` row) is the remaining +wiring step; until it lands the flag is inert and the overlay stays off. + +Three consequences for this module: `"computer_use"` MUST be present in +`_KNOWN_CONFIG_SECTIONS` (the guarded invariant that `to_dict()`'s emitted sections +equal that set); the dashboard's `_EDITABLE_CONFIG` exposes only the limits +(`computer_use.max_tree_nodes`, `computer_use.screenshot_max_px`) — never an +`enabled` key and never `allow_pointer_move`; and every numeric knob is clamped to +the same `*_LIMIT` ceiling the MCP tool schemas enforce, so a hand-edited +`config.json` cannot ask for an unbounded accessibility walk or a full-resolution +screenshot. + ### Security-Bounded Config Clamp Three resource-limit knobs are clamped to hard ceilings **at load time**, not just diff --git a/docs/system-specs/modules/governance.md b/docs/system-specs/modules/governance.md index 5cc48b97ffa..5cb881b02fb 100644 --- a/docs/system-specs/modules/governance.md +++ b/docs/system-specs/modules/governance.md @@ -38,7 +38,9 @@ can reorder strictness or redefine matching): - `_ORDINAL_SCALES`: `approval = yolo < auto < interactive`; `sandbox = off < standard < cc < strict` (verified against `sandbox.py`). - `_MATCHERS`: `identifier` (case-insensitive), `command` (case-sensitive - `fnmatchcase`), `path`, `host`, `mcp` (a `@server` grant covers `@server/tool`). + `fnmatchcase`), `path`, `host`, `mcp` (a `@server` grant covers `@server/tool`), + `bundle_id` and `cu_action` (both added for computer use — see + [Governed capability: computer use](#governed-capability-computer-use-native-desktop-gui-automation)). The `path` matcher normalizes **only the queried item** (`_norm_item`: expand `~`/`$VAR` → `os.path.abspath`, which anchors a relative path to the host CWD and collapses `.`/`..`) and matches it against the operator's pattern **expanded @@ -318,6 +320,23 @@ This is enforced solely by adding them to `security._SENSITIVE_HOME_DIRS` read+write gate across every surface. `assert_governance_paths_protected()` is a boot integrity check that fails closed if a refactor ever drops them. +**`~/.kiro/agents/*.json` and `~/.kiro/settings/mcp.json` are NOT on the floor +today** — an honest gap worth stating here because it bounds what the ceiling can +claim. Verified on the current tree: `is_sensitive_path("~/.kiro/agents/kirocrew.json")` +is `False` and `echo x > ~/.kiro/agents/kirocrew.json` is not blocked. Since +`hooks.on_tool_call` runs **only** from the `EVENT_PERMISSION_REQUEST` branch (the +`EVENT_TOOL_CALL` branch is documented informational-only — "the tool is already +running (auto-approved by kiro-cli). Hook results cannot block execution"), an +agent that writes itself into `allowedTools` makes kiro-cli stop sending +permission requests and **Plane A never runs at all** for that tool. This affects +every governed capability, not just computer use. Hardening those paths is +deliberately deferred to its own PR (it changes a path floor every legitimate +writer — `agent.py`, `handlers/mcp.py`, `handlers/agents.py`, `apps/backend.py` — +touches, and deserves review on its own merits). Until it lands, a capability +whose only enforcement is Plane A can be skipped this way. Computer use no longer +relies on either plane — it is not governed at all, and rests on the keystone enable +being agent-unwritable. See [computer-use.md](computer-use.md) → "Known limitations". + ## Profile resolution + binding A profile binds to a `surface` (cron/slack/dashboard/subagent/…), an `app` slug, @@ -331,6 +350,16 @@ canonical taxonomy parser — never re-implemented). Resolution is: (fail-closed, never a permissive fall-through), mirroring the dashboard `api_session_tool_policy` precedent. +**`identity_proven` is true for ANY non-empty session key**, so an unattended +surface that *does* carry a key — `cron:`, `subagent:`, `taskrunner` — +resolves to `None` (policy-ceiling-only), **not** `deny_all_profile`; only `_bg` +and `_hb` fall to deny-all. That is correct for most scopes and wrong for +computer use, which must not fall back to policy-only on a surface nobody is +watching a mouse on. Hence the feature-local unattended refusal in +`computer_use.gate` (a code rule that cannot be un-shipped by deleting a profile +file) plus the shipped `cu-off` profiles bound to those surfaces as the visible, +explainable form of the same decision. + **`host` surface (in-process host actions).** A governance check that is not driven by a user-facing surface — app activation (`apps.manager._app_activation_denied`), Slack workspace admission @@ -604,9 +633,14 @@ read-your-writes should add it deliberately, with its own tests. durable memory writes in `mcp_core._vet_memory_writes_governance` (at `learn_add`); script-hook execution in `hooks._script_hooks_capability_denied` (at `run_script_hook`); - app activation in `apps.manager._app_activation_denied` (at `enable_app`). All - route through the same `governance_permits` / `governance_floor_ordinal` - decision source. + app activation in `apps.manager._app_activation_denied` (at `enable_app`). + +Plane A carries **no live ordinal clamp**. It used to: a computer-use title under a +`computer_use.approval: interactive` floor had both auto-approve branches suppressed, +so the call fell through to interactive approval. That row and its clamp were removed +along with the rest of the computer-use governance model — see [Computer use is NOT +governed](#computer-use-is-not-governed-deliberately). The global `approval_mode` +row's live clamp remains reserved (see "Still-reserved in v1"). ## Foreign-agent import interaction @@ -852,9 +886,50 @@ denials leave the same forensic trail. profile. An app's own in-process tool calls (which carry `KIROCREW_APP_NAME`) do bind a per-app profile. App blast-radius is contained today by the `apps` activation allowlist + per-surface profiles. - -### Still-reserved in v1 - +- **Shell GUI automation is a `commands` item, never re-parsed.** `osascript`, + `cliclick`, `xdotool`, `ydotool`, `wtype`, `screencapture`, `scrot`, `grim`, + `import -window` and `nircmd` inside a Bash tool are governed by the + **`commands`** scope on the command body — no `computer_use.*` scope applies to + them, because a shell command is never decomposed into its GUI sub-effects. A + fleet banning computer use must also deny those `commands` patterns (see the + copy-pasteable fleet-ban policy below); a deny-mode `commands` pattern also + becomes an un-opt-out-able force-pin via `resolve_pinned_commands`. +- **The web terminal PTY is an ungoverned plane today.** + `dashboard/handlers/terminal.py` spawns a real PTY and contains **no** + `is_denied` / `is_sensitive_bash_command` / governance call, so + `screencapture` typed into it is bounded by neither the `commands` scope nor + any `computer_use.*` scope. It is an operator-only surface. Routing PTY input + through the same effective-deny floor as `on_tool_call` is tracked as its own + follow-up; do not describe computer-use governance as covering it. +- **Raster capture has two channels and neither is governed.** Computer use has no + `observations` scope any more, and Playwright's already-shipped + `browser_take_screenshot` never had one — a fleet that means "no raster capture" + must deny both `@kirocrew-computer` and `@playwright/browser_take_screenshot` via + the `mcp` scope. +- **The `mcp`-scope deny is now the ONLY governance lever over computer use, and it + is keyed on a renameable alias.** `mcp.deny: ["@kirocrew-computer"]` works on + unmodified shipped code, but the server key is derived by `mcp_server_alias()` from + an agent-mutable config: verified `mcp__kirocrew-computer2__click` and + `mcp__cu__click` both PERMIT under that deny. With the `capabilities.computer_use` + row removed there is no authoritative ban behind it — a fleet that must guarantee + the feature is off should not ship the keystone enable, and should treat the alias + deny as best-effort. See [Computer use is NOT + governed](#computer-use-is-not-governed-deliberately). +- **Cursor Motion has no governance row, and deliberately gets none.** The + fake-cursor desktop overlay (`computer_use/overlay*.py`) grants the agent + *nothing*: it draws an image, it does not move the pointer, it cannot deliver + input, and it is invisible to `screencapture` so it cannot even alter what the + model reads. It is a `config.json` display preference + (`computer_use.cursor_motion`, default OFF), and adding a scope for it would + imply an authorization decision where there is no capability to authorize. + The real pointer path (`click_method: "global"`, which warps the operator's + physical cursor) has no row either — it is reachable whenever the feature is on, + and is audited under its own SEL `tool_kind` rather than gated. +- **`kirocrew computer call` is subject to the same checks as an agent call.** The + CLI harness routes through the same `computer_use.tools.dispatch_tool` chokepoint, + so the keystone enable and the target policy apply to it, bound to the attended + `cli` surface (session key `cli_chat`). There is nothing governance-side left for a + policy author to bind to it. - **`approval_mode`** — the ordinal is parsed and **boot-floor-checked** (a profile looser than the policy mark aborts boot, like `sandbox.min_level`), but no approval chokepoint clamps the *live* approval pipeline through it yet: the @@ -864,6 +939,11 @@ denials leave the same forensic trail. genuinely-architectural follow-up (a single approval-policy resolution point fed by `governance_floor_ordinal("approval_mode")`). + There is no longer a second, live-clamped `approval` row to contrast this with: + `computer_use.approval` was removed with the rest of the computer-use governance + model, so `approval_mode` is once again the only row on the `approval` scale and + its live clamp is still the reserved half. + > **Capability `profile-absence` semantics (deliberate deviation from spec A.4 > rule 8).** The spec says a profile that OMITS a capability defaults it to > `false`. KiroCrew instead treats an omitted scope as *not governed by the @@ -1014,6 +1094,44 @@ contract (a stricter server is backward-compatible with consenting clients). expansion (longer length bound, per-turn injection, or richer pack tiers) — scope growth without server-recorded grants is not covered by this decision. +### Computer use is NOT governed (deliberately) + +Computer use (see [computer-use.md](computer-use.md)) has **no scope rows in +`SCOPE_CATALOG`** and no governance decision anywhere in its dispatch path. That is +a product decision, not an oversight, and it is a reversal: an earlier revision +shipped eight rows here (`capabilities.computer_use`, `computer_use.actions`, +`.apps`, `.app_names`, `.observations`, `.targets`, `.approval`, and +`capabilities.computer_use_pointer`) plus two custom matchers (`bundle_id`, +`cu_action`). All of it was removed. + +**What replaced it.** One operator opt-in on the keystone `computer_use.json`, +which `security._SENSITIVE_HOME_DIRS` fences the agent away from. The agent cannot +read or write that file, so it cannot enable its own desktop automation — and it +cannot drive KiroCrew's own window either (`computer_use/policy.py`), so it cannot +click the toggle in the UI. Those two facts are the entire boundary. + +**What this costs, stated plainly.** There is no way to express "computer use is +allowed but only for Preview", "read-only desktop access", "never type into a +password field" (beyond the always-on floor), or "every action must be approved" as +policy. A fleet that needs any of those should not enable the feature. The +`mcp` scope still works as a blunt instrument: denying `@kirocrew-computer` removes +the tools entirely, which is the one governance lever that remains. + +**If it is ever re-governed**, the rows belong back in this file's `SCOPE_CATALOG` +inline (never `register_scope()`d from the feature package): `load_security_policy()` +runs at boot before any feature import, and a policy naming an unregistered scope +raises "unknown governed key … (fail-closed)" — so a lazy registration would abort +boot on every governed host the day a fleet adds the row. + +Two things computer use still shares with this module, neither of them a decision: + +* `_CU_ACTION_CLASSES` — the code-owned `observe` / `mutate` / `pointer` / + `keyboard` / `text_entry` / `control` labels. `hooks` reads them for the + read-only auto-approve, and `gate.is_mutating_action` reads them so "which verbs + synthesize input" has one definition; +* `CU_MCP_SERVER` / `is_computer_use_title` — the server key and title prefix, used + by `classify_tool_title` to route a computer-use title to the ordinary `mcp` pair. + ## Audit `sel.log_governance_decision` records a `governance_decision` event @@ -1029,6 +1147,10 @@ read-only operator diagnostics. `show` reports the ceiling's **proven** provenan issuer string. `explain` traces the rule/layer/reason and the live gate verdict. Deliberately **not** exposed as an MCP tool: it surfaces governance internals that the agent (the governed subject) should not enumerate. +(The two `validate` warnings that used to be listed here were specific to the +computer-use `bundle_id` matcher and the `capabilities.computer_use` row, both of +which are gone.) + ## Companion (separate package, separate CR) The `amazon` companion contributes the restrictive posture as its @@ -1053,7 +1175,9 @@ carve-out stay as code. It expects `CONTRACT_VERSION == 1` (pinned pre-launch). `GOVERNANCE_ERROR_REASON` (the eval-error marker consumers match on), `vet_and_audit`. - `security.py` — `_SENSITIVE_HOME_DIRS` keystone entries. -- `hooks.py` — Plane A gate threading. +- `hooks.py` — Plane A gate threading + the computer-use read-only auto-approve + (`_cu_read_only_auto_approve`, which reads the action-class table rather than a + governance row). - `sel.py` — `log_governance_decision`. - chokepoints: `sandbox.py`, `mcp_cron.py`, `subagent.py`, `mcp_core.py`. - `messaging/identity.py` — `channel_inbound_permitted` (the per-message inbound @@ -1063,6 +1187,9 @@ carve-out stay as code. It expects `CONTRACT_VERSION == 1` (pinned pre-launch). - `dashboard/handlers_system.py` — `GET /api/governance/channels`. - `dashboard/handlers/security.py` — `GET /api/governance/policy` (posture-only serialization). +- chokepoints: `sandbox.py`, `mcp_cron.py`, `subagent.py`, `mcp_core.py`, + `computer_use/gate.py` (`require_computer_use` fail-closed + + `apply_observation_ceiling`). - `cli.py` / `cli_commands.py` — the `policy` command. ## Tests @@ -1081,4 +1208,5 @@ the per-transport inbound gates), `test_governance_channels_endpoint.py` `test_governance_policy_viewer.py` (`/api/governance/policy` posture-only, incl. `test_detail_never_leaks_rule_contents`), `test_governance_updates.py` (the `updates` pins, the shared seam's fail-open-on-error disposition, and the -tracked-remote resolution). +tracked-remote resolution), and `test_computer_use_gate.py` (that the +computer-use gate is audit-only and permits — see the section above). diff --git a/docs/system-specs/modules/security.md b/docs/system-specs/modules/security.md index 12f1ff970aa..3687e83a037 100644 --- a/docs/system-specs/modules/security.md +++ b/docs/system-specs/modules/security.md @@ -139,6 +139,8 @@ under `(allow default)`, never an edition-resolved or user-writable executable. - **App admission trust root (keystone invariant)** — `~/.kiro/crew/app_admission.json` (the sole fleet-controlled App Kit admission ceiling loaded by `apps/admission.py`) is on the sensitive floor, read- and write-blocked through the shared file gate. Because an **absent** policy falls open (admit), a governed agent that could `fs_write`/`tee`/`rm` this file could delete or neuter its own admission ceiling and re-admit a banned/unsigned app. Adding it to `_SENSITIVE_HOME_DIRS` also arms the bash write/extract matcher (`tee`/`rm`/redirect). The gateway's own loader (`apps.admission.load_app_admission_policy`) opens it directly via `path.read_text()` and does **not** route through `is_sensitive_path()`, so legitimate policy loading is unaffected. - **Runtime exec dir trust root (run-marker sandbox-escape)** — the `run/` dir under the crew home is on the sensitive floor (under every `_CREW_HOME_PREFIXES` entry) because the gateway executes its contents **outside** the agent sandbox: the sandbox launcher scripts (`sandbox.py` execs `python /run/kirocrew_sandbox_*.py`) and the remote-instance run-marker `gateway-.bin` (`instances/run_marker.py`), whose recorded launcher path the SSH token-mint **and** the `restart_remote` action read and `exec` on the remote host. Without this floor entry a prompt-injected / sandboxed agent that could `fs_write` (or `tee`/redirect) a marker — or a launcher — into `run/` could point it at an attacker-controlled binary that the next routine token refresh or restart execs unsandboxed (owner + `-x` checks don't help; agent writes run as the same user) — a reachable sandbox escape. Classifying the whole dir also arms the bash write/extract matcher. The gateway's own writers (`instances/run_marker.write_marker` via `atomic_write`, and the sandbox launcher writer) open these paths directly and do **not** route through `is_sensitive_path()`, so gateway startup/spawn is unaffected. +- **Computer-use primary enable (keystone leaf `computer_use.json`)** — the on/off switch for native desktop GUI automation (see [computer-use.md](computer-use.md)) is `~/.kiro/crew/computer_use.json`, added to `_CREW_SECRET_LEAVES` so it is read+write-blocked under every `_CREW_HOME_PREFIXES` entry, on both the tool path (`is_sensitive_path`) and every shell form (`is_sensitive_bash_command` — `cat`, `>`, `tee`, `rm`, plus `tar -C` / `unzip -d` extraction into the trust root via `_EXTRACT_INTO_TRUST_ROOT_RE`). **It is deliberately NOT in `config.json`**, and the precedent is the denied-command opt-out immediately below: `is_sensitive_write_path("~/.kiro/crew/config.json")` is `True`, but `is_sensitive_bash_command("echo x > ~/.kiro/crew/config.json")` is `None` and `is_denied(...)` is `None` (`_WRITE_PROTECTED_BASH_LEAVES` is `('.data-home-ready',)` only), so a `config.json` toggle would be flippable by a prompt-injected agent through any redirect. A primary enable for full desktop observation plus input synthesis is a **security ceiling**, the same class as the deny opt-out, so it lives on the keystone. Reads fail soft to `{}` → **disabled**, and `is_enabled()` is a strict identity test against `True` (a hand-edited `"enabled": "false"` or `1` does not enable desktop control). The only writer is the dashboard PUT handler, which does not route through the agent tool gate; `enable_state.load_state()` opens the file directly, so legitimate reads are unaffected. The same file also carries `allow_pointer_move` — the opt-in for the one click path that warps the operator's REAL mouse pointer (`click_method: "global"`) — so that flag inherits the identical protection with no new leaf: it is read with the same strict `is True` identity test, and it is only half the gate (the `capabilities.computer_use_pointer` governance row is the other half, and neither substitutes for the other). + **Write-only config protection** (`is_sensitive_write_path` in `security.py` + `hooks.py`) — runtime config files are protected against *modification* by agent tools while staying *readable*: - `~/.kiro/crew/config.json` and `~/.kiro/crew/config.local.json` are in a write-only tier (`_WRITE_PROTECTED_HOME_PATHS`, expanded under every `_CREW_HOME_PREFIXES` entry so the pre-move legacy copy is covered too), deliberately NOT in the read+write `_SENSITIVE_HOME_DIRS` list above — the dashboard file viewer, `cat`, and knowledge indexing legitimately read config. - `is_sensitive_write_path(path)` is a superset of `is_sensitive_path(path)`, sharing the same `_path_in_home_dirs` resolve/casefold core so the two gates can't drift. `hooks.on_tool_call` denies a file-EDIT tool call (ACP `edit` kind) whose `path`/`file_path` resolves to a config file. @@ -512,6 +514,92 @@ redirects/substitution/backgrounding); for a non-shell tool when `tool_kind in {"read", "fetch"}` or `slack.gateway._is_read_only_tool(tool_name)` is True. Both classifiers are imported function-locally to avoid an import cycle. +Computer-use observation tools get their own **explicit** pair in that same +branch (`_cu_read_only_auto_approve`), keyed on the code-owned +`governance.computer_use_action_classes()` table rather than the +`_is_read_only_tool` title heuristic — that heuristic keys on a leading verb, and +an agent-supplied title must never decide whether a keystroke is synthesized into +somebody's window. It is additionally gated on the keystone primary enable, so no +auto-approval can exist while the feature is off. Immediately above it, +`_cu_approval_floor_forces_prompt` implements the `computer_use.approval` ordinal +clamp by suppressing BOTH auto-approve branches when a policy sets the floor to +`interactive`; see [governance.md](governance.md). + +### Computer use: a pixel/AX surface the path matchers cannot see + +Native desktop GUI automation ([computer-use.md](computer-use.md)) is a security +surface unlike every other one in this module, and the difference is worth stating +plainly: **`is_sensitive_path` cannot see it.** A click has no path, a keystroke +has no command body, and a window's pixels have no filename. So none of the +mature matchers in this document apply — not the sensitive-dir regex, not the +relative-traversal block, not the symlink resolution, not the write-protected +tier. A Terminal window rendering `~/.aws/credentials` is, to the path gate, no +different from a text editor. + +Three controls carry the weight instead: + +1. **The app denylist is a floor in code** (`computer_use/policy.py::_DENIED_BUNDLE_PREFIXES`), + matching by bundle-id PREFIX (so a helper process under a blocked bundle is + covered) or case-insensitive process-name SUBSTRING (the Windows/Linux drivers + may only ever learn a process name). Five categories: `terminal`, + `credential_manager`, `system_settings`, `auth_prompt`, `kirocrew_self`. + **`terminal` is refused for ALL verbs, reads included** — for two independent + and individually sufficient reasons: a terminal window renders whatever the + operator last `cat`ted as both AX text and pixels (and redaction only catches + shapes it knows — it will not recognise an SSH private-key body or an opaque + bearer token), and a keystroke into a shell prompt IS arbitrary command + execution that never passes `hooks.on_tool_call`, bypassing the entire + denied-command ceiling, the sensitive-path gate and the SEL audit of bash + calls. `kirocrew_self` is there because KiroCrew's own dashboard can flip this + feature's primary enable and the denied-command opt-out — controls that are + out-of-band precisely so the agent cannot reach them. The list is + operator-EXTENSIBLE (`extra_denied_apps` can only ADD) and never + operator-shrinkable; the governance `computer_use.apps` ruleset is the + enterprise force-pin on top. +2. **The secure-SUBROLE check**, and it must be the subrole. A real macOS password + box reports `AXRole = "AXTextField"` (innocuous) with + `AXSubrole = "AXSecureTextField"` and a **readable** `AXValue` — live-verified. + So the intuitive `AXRole == "AXSecureTextField"` check **misses every password + field**. The driver sets `secure = (role == SECURE_SUBROLE or subrole == + SECURE_SUBROLE)` and three protections key off that one flag: the renderer + emits `` for the value (never the bytes, not truncated, not + masked-with-a-hint), `policy.check_input_target` refuses + `set_value`/`type_text`/`press_key` at a secure target, and a window containing + ANY secure node gets **no screenshot at all** (whole-window suppression — there + is no reliable way to blank a sub-rectangle of an already-encoded JPEG, and a + partial redaction that missed would be worse than none). This floor has **no + policy key and none will be added**: `resolve(None, None, …)` permits + everything on an ungoverned host, so anything expressed only as a governance + scope leaks by default for every single-user install. It belongs with + `_SENSITIVE_HOME_DIRS` and the AKIA redaction, not with governance. +3. **The input-text scan as an explicit SECOND layer**, not the primary control. + Text bound for another app's window is run through `is_sensitive_bash_command` → + `audit_bash_exfiltration` → `is_denied` (called with `denied_regexes=None`, so + it fails closed to the full built-in rule set — a user's opt-out from a bash + deny rule is a decision about commands the AGENT runs under the tool gate, not + a licence to type the same command into somebody else's window). This module + already records the maintainers' position that chasing shell-parser + completeness in a text matcher is a losing game, which is exactly why "refuse + the app wholesale" comes first. + +**Accepted residual — the screenshot directory stays agent-readable.** Persisted +JPEGs live in `/kirocrew-computer-shots`, created `mode=0o700` with each file +passed through `platform_compat.restrict_to_owner` and ring-trimmed to 200 — but +the agent can still reach them with `fs_read`. This is the same posture browse +already ships; computer use widens WHAT can be in the frame (any window, not one +browser tab), which is bounded by per-window capture only (never full-screen) and +the whole-window suppression above. The design does not widen the posture and +does not claim to close it. A reviewer will find this independently, so it is +recorded here rather than left implicit. + +Two further boundaries this module does **not** cover, stated so nobody assumes +otherwise: shell GUI automation (`osascript`, `cliclick`, `xdotool`, +`screencapture`, …) is a `commands`-scope item governed by the deny floor, never +re-parsed into GUI sub-effects; and the **web terminal PTY** +(`dashboard/handlers/terminal.py`) contains no `is_denied` / +`is_sensitive_bash_command` / governance call at all, so it is an operator-only, +ungoverned plane today. + ### Suspicious Bash Patterns (`security.py`) 55 patterns in `SUSPICIOUS_BASH_PATTERNS` checked by `audit_bash_command()` at tool invocation time. Patterns with `*` use `fnmatch` glob matching; others use substring matching. diff --git a/scripts/scrub-allowlist.txt b/scripts/scrub-allowlist.txt index cddb88cc52e..1b3c7f56286 100644 --- a/scripts/scrub-allowlist.txt +++ b/scripts/scrub-allowlist.txt @@ -98,6 +98,13 @@ packaging/signing/manifest-template.json:.*com\.amazon\.kiro\.crew packaging/signing/sign-dmg.sh:.*com\.amazon\.kiro\.crew packaging/build-desktop.sh:.*com\.amazon\.kiro\.crew docs/signing-runbook.md:.*com\.amazon\.kiro\.crew +# Same signed identity, in the computer-use self-denylist: KiroCrew's own +# dashboard can flip this feature's primary enable, edit the denied-command +# opt-out and change approval mode — all reachable ONLY out-of-band precisely so +# the agent cannot reach them. Refusing our own bundle as an automation target is +# what stops the agent from clicking its way around those keystone protections, +# so the id must be matched literally. Anchored to the one file that needs it. +src/kiro_crew/computer_use/policy.py:.*com\.amazon\.kiro\.crew # Public AWS documentation URLs (docs.aws.amazon.com / aws.amazon.com) — generic # AWS product references, not internal couplings. diff --git a/src/kiro_crew/agent.py b/src/kiro_crew/agent.py index fb72136868c..40e5258cb83 100644 --- a/src/kiro_crew/agent.py +++ b/src/kiro_crew/agent.py @@ -48,11 +48,13 @@ from kiro_crew.browser.setup import converge_playwright_servers from kiro_crew.config import config_dir from kiro_crew.config import config_path as _mc_config_path +from kiro_crew.config.paths import _valid_override_home from kiro_crew.env import augmented_path from kiro_crew.mcp_utils import mcp_server_alias from kiro_crew.platform import current_context from kiro_crew.platform import redact_via_context as redact from kiro_crew.platform import safe_context_call +from kiro_crew.platform.governance import CU_MCP_SERVER from kiro_crew.security import is_sensitive_path from kiro_crew.sel import ( # circular import: sel imports config which imports agent SecurityEvent, @@ -286,6 +288,31 @@ def _usable(p: str | Path) -> bool: return "kirocrew" +def _managed_mcp_env() -> dict[str, str]: + """Env every managed KiroCrew MCP server is launched with. + + Pins ``KIROCREW_HOME`` when the gateway is running under an override, because + a child process does NOT inherit it: the spec's ``env`` is the only channel. + Without this the gateway and its own stdio shims read DIFFERENT data homes, + which is silent and self-contradictory rather than merely wrong — + ``computer_use.json`` is written to the override home by Settings while + ``mcp_computer`` reads the DEFAULT home, so the panel shows the feature ON + while the shim publishes an empty ``tools/list`` and the agent truthfully + reports it has no computer-use tools. The same split would desynchronise the + cron store and the lessons file. + + Resolved through ``_valid_override_home`` rather than reading the env var + directly, so an override the loader REFUSES (a filesystem root, ``/usr``) is + not propagated to children that would then disagree with the gateway in the + other direction. + + Returns ``{}`` on a default install, which keeps the emitted spec + byte-for-byte what it is today (``_prune_empty`` drops an empty ``env``). + """ + override = _valid_override_home() + return {"KIROCREW_HOME": str(override)} if override else {} + + def _kirocrew_mcp_invocation(subcommand: str) -> tuple[str, list[str]]: """Resolve a CWD- and shebang-independent invocation for a built-in MCP server (``kirocrew-cron`` / ``kirocrew-core``). @@ -319,6 +346,18 @@ def _kirocrew_mcp_invocation(subcommand: str) -> tuple[str, list[str]]: _MANAGED_MCP_SERVERS: dict[str, dict] = { "kirocrew-cron": {"invocation_fn": lambda: _kirocrew_mcp_invocation("mcp-cron")}, "kirocrew-core": {"invocation_fn": lambda: _kirocrew_mcp_invocation("mcp-core")}, + # Computer use (native desktop GUI automation). Registered unconditionally — + # its stdio shim returns an EMPTY tools/list while the keystone primary enable + # is off, so a disabled feature costs the model no context and needs no + # per-server ``enabled_fn`` in this loop. + # + # DELIBERATELY NO ``autoApprove`` KEY, and none may ever be added: kiro-cli + # approves an autoApproved MCP tool locally and emits no permission request, + # so ``hooks.on_tool_call`` — the PreToolUse gate carrying the always-on deny + # floor, the sensitive-path check and the governance ceiling — is NEVER + # reached for it. For a tool that can click in an already-authenticated + # application that would be a complete gate bypass. + "kirocrew-computer": {"invocation_fn": lambda: _kirocrew_mcp_invocation("mcp-computer")}, } @@ -1348,6 +1387,12 @@ def build_agent_config() -> dict: cmd = spec.get("command") or spec["command_fn"]() args = list(spec["args"]) entry = {"command": cmd, "args": args} + # Pin the data home so the shim cannot read a DIFFERENT one than the + # gateway that spawned it (see _managed_mcp_env). Omitted entirely on a + # default install, so the emitted spec is unchanged there. + env = _managed_mcp_env() + if env: + entry["env"] = env if "autoApprove" in spec: entry["autoApprove"] = list(spec["autoApprove"]) mcp[name] = entry @@ -1393,6 +1438,21 @@ def _refresh_dynamic_fields(config: dict) -> None: # the downstream stdio-force in cc_agent / acp.client.) entry.pop("url", None) entry.pop("headers", None) + # Data-home pin — refreshed like command/args rather than preserved like + # ``autoApprove``, because it is OURS, not a user customization: it must + # track the home the gateway is actually running under. A config written + # under an override and later refreshed on a default install would + # otherwise keep pointing the shims at the stale home. Merged into any + # existing ``env`` so a user's own variables survive, and the key is + # REMOVED (not left stale) when there is no override. + pinned = _managed_mcp_env() + env = dict(entry.get("env") or {}) + env.pop("KIROCREW_HOME", None) + env.update(pinned) + if env: + entry["env"] = env + else: + entry.pop("env", None) # Seed autoApprove only for genuinely new entries; if the user # deliberately removed autoApprove from an existing entry we # must not re-add it on every refresh. @@ -1966,6 +2026,48 @@ def _resolve_command(cmd: str, env: dict | None) -> str | None: resources=f"{', '.join(added_refs)} added to tools (fresh install)", ) + # Narrow ADD-only exception on EXISTING configs, mirroring the + # ``tool_search`` precedent in _refresh_dynamic_fields: ensure the + # computer-use @ref is in ``tools``. + # + # Without this, an UPGRADING install never gains the ref — the fresh-install + # branch above is the only place it is added — so ``kirocrew-computer`` is + # registered in ``mcpServers`` but kiro-cli exposes none of its tools, and the + # feature silently does nothing for every pre-existing user. (Unlike a + # third-party MCP, the user cannot have "opted out" of a ref that never + # existed on their install.) + # + # DELIBERATELY tools-only, never ``allowedTools``: that list is kiro-cli's + # blanket auto-approve, and an auto-approved MCP tool is approved locally by + # kiro-cli — it emits no permission request, so ``hooks.on_tool_call`` (the + # deny floor + governance ceiling + approval clamp) is never reached for it. + # Granting it here would delete the PreToolUse plane for a tool that can click + # and type into an already-authenticated application. + # + # Gated on the shipped template actually granting the ref (so an edition that + # drops computer use is respected) and on the server having resolved, and + # scoped to this ONE server so no other managed ref is re-added behind the + # user's back. The primary enable still lives in the keystone file, so a config + # that gains the ref is not a feature that turns itself on: the shim answers an + # empty tools/list until the user opts in from Settings. + if not fresh_install and CU_MCP_SERVER in valid_servers: + cu_ref = f"@{CU_MCP_SERVER}" + shipped_tools = get_shipped_tools().get("tools", []) + existing_tools = config.get("tools") + if ( + isinstance(existing_tools, list) + and cu_ref in shipped_tools + and cu_ref not in existing_tools + ): + existing_tools.append(cu_ref) + sel().log_api_access( + caller="system", + operation="mcp_tools_added", + outcome="ok", + source="install_agent", + resources=f"{cu_ref} added to tools (existing config upgrade)", + ) + # Final dedup (preserves order). for key in ("tools", "allowedTools"): config[key] = list(dict.fromkeys(config.get(key, []))) diff --git a/src/kiro_crew/builtin_skills/computer-use/SKILL.md b/src/kiro_crew/builtin_skills/computer-use/SKILL.md new file mode 100644 index 00000000000..4c3a50d4dd9 --- /dev/null +++ b/src/kiro_crew/builtin_skills/computer-use/SKILL.md @@ -0,0 +1,371 @@ +--- +name: computer-use +description: Read and drive native desktop applications through the accessibility layer — list on-screen apps, snapshot one window as a numbered element tree, then click / type / set a value / scroll / drag / run a named action, by element index or by screen coordinates. Use for work in a desktop app rather than a web page. macOS only; off unless the user enabled it in Settings. +triggers: desktop, desktop app, native app, app window, on screen, click button, type into, accessibility, a11y, AXUIElement, computer use, drive the app, Finder, Preview, TextEdit, Excel, Word, System Events, !browser, !web page, !playwright +--- + +# Computer Use — driving native desktop apps + +You have MCP tools that read and operate the **user's real applications** through +the operating system's accessibility layer. This is not a browser: use it when the +work lives in a desktop app (a spreadsheet, a PDF viewer, a native internal tool, +a dialog box), and use the Playwright `browser_*` tools for web pages. + +Two things to internalise before your first call: + +- **Address elements by index, from a snapshot you were just shown.** That is the + path to prefer for everything: it activates the control directly, it is checked + against drift, and the mouse pointer does not move. Coordinates exist as a + fallback for canvases, maps and custom-drawn UI that expose no element — see + [Coordinates and dragging](#coordinates-and-dragging-the-fallback-not-the-default). +- **It is off unless the user turned it on** (Settings → Computer Use, macOS only). + A refusal saying so is a real configuration answer, not a transient error — + relay it and stop; do not retry. + +## The loop + +**1. Find the app.** + +``` +computer_list_apps() +``` + +Returns the on-screen applications with their bundle ids, pids and window titles. +Skip this if the user named an app you can pass straight through — `app` accepts a +display name (`"Finder"`, `"Preview"`) or a bundle id +(`"com.apple.finder"`), matched case-insensitively. + +**2. Snapshot the window. Do this FIRST, every turn — with a screenshot.** + +``` +computer_get_state(app="Finder", screenshot=True) +``` + +**Do NOT pass `screenshot=false` here.** Omitting the argument already captures one +(the operator's Settings default), and that capture is what opens the user's **live +view**: the dashboard mirrors the JPEG into a floating panel, which only appears once +a frame exists. Turning it off on the first call leaves the user watching a blank +space while you drive their machine. Passing `screenshot=True` explicitly is fine and +harmless if you want to be sure. + +The capture is not for your benefit — you get a file PATH, not an image, and you +should keep reading the outline. One frame opens the panel; it stays open for the rest +of the task. + +Optional: `text_limit` (per-element text cap), `max_tree_nodes`, `max_tree_depth`, +`screenshot` (bool). You get a numbered outline: + +``` +App=com.apple.finder (pid 1041) +Window: "Documents", App: Finder. + +0 window "Documents" @ x=0,y=0 900x600 + 1 splitgroup + 2 scrollarea + 3 button "Back" [AXPress] @ x=18,y=12 28x24 + 4 textfield "report" (editable) @ x=120,y=52 300x24 + 5 row "Q3 numbers.xlsx" (selected) @ x=8,y=90 880x20 + 7 textfield + +Window origin on screen: x=220,y=118 (900x600). Element frames above are +relative to it — add the origin for a screen point. +Focus: element 4 (AXTextField "report"). +Selected text: [Q3] +``` + +Reading one line: the number at the start is the `element_index` you pass to +every action. `[AXPress]` and friends are the actions that element advertises. +The indentation is containment, so you can tell a toolbar button from a table +cell. Then, in order: + +- **`(editable)`, `(selected)`, `(expanded)`, `(disabled)`** — state the role + cannot carry. `(editable)` is the one to check before typing: a read-only + field looks identical to a writable one otherwise, and typing into it + succeeds while the text goes nowhere. If a text field has no `(editable)`, + it will not accept input — find the one that does instead of retrying. +- **``** — the caret is here. "Type this in" means this element, and + `computer_type_text` without an `element_index` goes here. +- **`@ x=…,y=… WxH`** — position and size, **relative to the window**, in + pixels. Absent when the element exposes no geometry (ordinary) or the window + rect could not be read. + +The trailing lines: the **window origin** is what converts a frame to a screen +point (`computer_click(x=…, y=…)` takes SCREEN coordinates — add the origin, do +not pass a frame straight through); **Focus** names the focused element; and +**Selected text** is what the user has highlighted, which is what a request like +"rewrite what I selected" refers to. + +A `` element shows no title, no value, no traits and no frame — only +that it exists. + +**3. Act by index.** + +| Tool | Use it for | +|---|---| +| `computer_click(app, element_index)` | press a button, checkbox, menu item, link, row | +| `computer_type_text(app, element_index, text)` | type text into that element. `element_index` is **required** — see the note below | +| `computer_set_value(app, element_index, value)` | replace a field's whole contents in one step | +| `computer_press_key(app, element_index, key)` | a key or chord — `"return"`, `"tab"`, `"escape"`, `"cmd+s"`, `"cmd+shift+a"`. **Paste (`cmd+v`) is refused** — the clipboard cannot be inspected, so use `computer_type_text` with the literal text | +| `computer_scroll(app, element_index, direction, pages?)` | scroll a scrollable area (`up`/`down`/`left`/`right`) | +| `computer_perform_action(app, element_index, action)` | run one of the element's own advertised actions when nothing above fits | +| `computer_click(app, x, y)` | click a point when the target has no element — see below | +| `computer_drag(app, from_x, from_y, to_x, to_y)` | a canvas stroke, a slider sweep, a range selection, a reorder | + +**Every one of these needs an `element_index`, and the keyboard tools are the ones +to remember.** There is no "type into whatever is focused" form: an unnamed target +has no role or subrole, so the secure-field check cannot inspect it, and an indexless +keystroke would land in a focused password box. That applies to `computer_press_key` +too — `press_key("tab")` can *move* focus onto a password field, and the next +keystroke would go there. So name the field you mean; if you want to tab through a +form, address each field by index instead of tabbing blind. `computer_click` is the +only exception, and only because it takes coordinates as the alternative. + +Every action returns a **refreshed** tree, so after a click you already have the +new indices — do not call `computer_get_state` again just to re-read them. The same +goes for window position and size: they are in the snapshot header you were already +shown. **Re-probing something the last response already told you is the most common +wasted turn.** + +**Refresh the user's view as you go.** Action results carry the structure but no +pixels, so the live-view panel freezes on your last screenshot while you work. After +a step that visibly changes the screen — a window opened, a dialog appeared, a file +saved, a page navigated — call `computer_get_state(app=…, screenshot=True)` once to +push a fresh frame. Use judgement rather than doing it after every keystroke: typing +five fields is ONE visible change, not five, and each screenshot costs time and +tokens. The test is "would the user see something different now?", not "did I just +call a tool?". + +### Coordinates and dragging: the fallback, not the default + +`computer_click` takes **either** `element_index` **or** both `x` and `y` — never +both and never neither; supplying both is refused, because the two name different +targets and there is no rule for which should win. + +Reach for coordinates only when the outline has nothing to address: a drawing +canvas, a map, a timeline, a chart, a custom-drawn control. An element index is +better whenever one exists — it is verified against UI drift, and a coordinate is +delivered to whatever happens to be at that point *now*, so a window that reflowed +after your snapshot will take the click somewhere you did not mean. Re-snapshot +before a coordinate click if anything has changed. + +Optional on both: `mouse_button` (`left`/`right`/`middle`), `click_count` (1-3 for +single/double/triple), and `click_method`: + +- **`auto`** (the default) — element index → the accessibility press; coordinates → + an app-scoped mouse event. Correct almost always; do not override it without a + reason. `auto` will **never** pick the pointer-moving path, so leaving it alone is + always the pointer-safe choice. +- **`app_post`** — send the click to the target app at that point *without moving the + user's mouse*. This is what makes clicking a background window safe. +- **`accessibility`** — force the press path; requires `element_index`. +- **`global`** — **moves the user's real mouse pointer** and clicks there. You have + to ask for it by name; nothing resolves to it implicitly — that naming + requirement is the only thing between an ordinary click and the user's cursor. + Every use is separately recorded in the audit log. Ask for it only + when a click has to be physically real (a Dock item, a menu-bar extra, UI that + ignores posted events) — say so in your reply *before* you use it, because the + user's cursor will jump out from under their hand, and never use it as a + first attempt or a retry after an ordinary click failed for some other reason. If + it is refused, do not retry: use `app_post` or find another route. + +- **`sky_click`** — clicks a window that is **behind other windows**, without + raising it and without moving the pointer. Ask for it by name when `app_post` + reached the app but nothing happened. That is the signature of an app whose + renderer does its own hit-testing and so ignores a posted click while another + window is in front — browser-based and iPad-ported apps behave this way (Chrome, + VS Code, Slack, Freeform are the ones you will meet). It is the + one method built on a private Apple API, so it can stop working on a future macOS — + when it is unavailable the refusal says so and names `app_post`. Do not reach for it + first; reach for it when a covered window is the reason a click did nothing. + +`computer_drag` is coordinate-only — no accessibility action expresses a sweep +between two points — and it takes the same `mouse_button` / `click_method` options. + +**4. Release when you are finished with the app.** + +``` +computer_end_turn() +``` + +Drops the cached snapshots. Call it when the desktop part of the task is done. It +is cheap and it prevents a stale-index refusal later in the conversation. + +## A screenshot has TWO purposes — keep them straight + +This trips models up, so be explicit about which one you are serving: + +1. **The user's live view (usually why you want one).** Capturing a screenshot is + what makes the floating panel appear and update, so the user can watch you work. + This costs you almost nothing: you get a file PATH, not an image, and you do not + read it. **Ask for it on your first snapshot and after each visible change.** +2. **Your own perception (rarely).** Actually READING the file costs ~8,000 tokens + and is a last resort — the outline is your channel. + +So "take a screenshot" and "look at a screenshot" are different acts. Do the first +liberally; do the second only when the tree genuinely cannot answer the question. + +## When something does not work: change the MECHANISM, not the arguments + +This is the rule that separates a two-call fix from a twenty-call loop. When an +action fails or nothing visibly changed, ask yourself one question before the next +call: + +> **Am I varying arguments on the same mechanism, or switching to a genuinely +> different one?** + +Nudging a coordinate by 20px, retrying the same `element_index`, or re-issuing the +same `click_method` are all the SAME mechanism. **Two failures on one mechanism is +the signal to switch — not a reason to try a third variation.** + +The ladder, in order. Go down one rung per failure; never repeat a rung: + +1. **`element_index`** on the control itself (the default, and right ~90% of the time) +2. **A different element** — the row or cell that CONTAINS your target, or the + control's parent; icon-only buttons often only respond one level up +3. **`computer_perform_action`** with an action the element actually advertises — + read the `[AXPress]`-style list in the outline rather than assuming +4. **Keyboard** — `computer_press_key`. Menus, dropdowns, date pickers and + scrollbars are frequently keyboard-reachable when they are click-hostile +5. **Coordinates** — `x`/`y` from the element's own position, not from the screenshot +6. **`click_method: "sky_click"`** if the window is covered by another window +7. **Stop and tell the user what you tried.** Two sentences, naming the rungs. That + is a better outcome than a twentieth call + +**Never** re-run a rung that already failed, and never escalate preemptively — do not +reason "this is Electron, so clicking will fail, so I will start at coordinates." +React to an observed failure, do not predict one. + +## Do not report success you have not observed + +An action that returned without an error is **not** proof it landed. The refreshed +tree that comes back with every action is your evidence — read it and name what +changed: a new value, a dialog that appeared, a menu that closed, a button that +became disabled. + +- **If nothing in the tree changed, the action probably did nothing.** Say so and go + down the ladder. Do not report success. +- **A target that VANISHED is usually success, not failure.** A button that is gone + after you clicked it, a dialog that closed, a row that disappeared after delete — + the element being unfindable is the expected outcome. Do not "retry" it. +- Do not go looking for evidence that cannot exist: the Cursor Motion overlay is + invisible to screenshots, so its absence proves nothing. + +The most common failure in desktop automation is reporting success on a silently +dropped action. The second most common is retrying an action that already worked. + +## Prefer the outline; read the screenshot only if you must + +The tree is the primary channel and it is usually sufficient. When a screenshot is +attached you get a **path**, not an image: + +``` +Screenshot: /var/folders/.../kirocrew-computer-shots/shot-1769472013411.jpeg + (1280x604 jpeg, 24.2 KB) — read it with the fs_read tool only if the tree is + insufficient. +``` + +Open it with the file-read tool **only** when the outline genuinely cannot answer +the question — a chart, a rendered document, a layout problem, or a control the +accessibility layer did not expose. Reading it costs roughly 8,000 tokens. If the +user asked "show me", just give them the path; the dashboard renders it. + +Pass `screenshot=false` only when nobody is watching and you purely need +structure — a long mechanical loop over many elements, for instance. Prefer leaving +it on: the cost of capturing (not reading) one is small, and it is what keeps the +user's live view alive. + +## Reading the refusals correctly + +These are **answers**, not failures. Relay them and adapt; do not loop. + +| You see | What it means | What to do | +|---|---|---| +| a value ending in `…` | the text was cut at `text_limit`, not truncated by the app | re-snapshot with a larger `text_limit`; do NOT read the screenshot to recover it, and do not tell the user the content is missing | +| `[tree truncated at N nodes]` | the window has more controls than the budget | raise `max_tree_nodes`, or scroll to bring your target into range — the rest of the window is real, you just have not been shown it | +| `no state for 'X'. Call computer_get_state first.` | you acted without a snapshot | snapshot, then act | +| `state for 'X' is 214s old. Call computer_get_state again.` | the snapshot expired (90s) | snapshot again | +| `element_index 7 changed since the last computer_get_state (was 'AXButton "Save"', now 'AXButton "Delete"')` | the UI moved under you — this refusal is what stopped you clicking the wrong thing | snapshot again and re-locate the element by its label, not its old number | +| `'…' is a blocked target for computer use (…)` | KiroCrew's own dashboard is permanently refused — driving it would let you change your own security settings | do the task another way; tell the user why | +| `refusing to type this text into 'X': …` | the text looked like a sensitive command or credential | do not rephrase to get around it; explain and stop | +| `refusing to … a secure text field` | the target is a password field | ask the user to type it themselves | +| `computer use is disabled …` | the primary switch is off | tell the user to enable it in Settings → Computer Use; do not retry | +| `computer use is not supported on this platform (…)` | not macOS | say so once | +| `moving the real mouse pointer is switched off for this caller` | you asked for `click_method: "global"` on a leg that refuses it | use `app_post` (or an element index) — neither moves the pointer | +| `give either element_index or both x and y, not both forms` | you supplied two different targets in one call | pick one — the element index if the outline has the control | + +## When the tree is lying to you + +Only about a third of macOS apps implement accessibility well, so a tree that looks +authoritative can be wrong. Recognising this is what stops you clicking the same +wrong index four times: + +- **Repeated or empty labels.** Three rows all reading `row` with no title, or a + blank `textfield` where you expected "Search" — the tree cannot disambiguate them. + Use position, or use the containing element. +- **A near-empty tree from an app that clearly has content.** Electron apps (Slack, + VS Code, Obsidian, Freeform) need an accessibility opt-in that takes ~2s; the first + snapshot can be a 3-node stub. Snapshot once more before concluding anything. +- **The tree disagrees with the screenshot.** Trust the screenshot about what EXISTS + and the tree about what is ADDRESSABLE. If a control is visible in the image but + absent from the outline, it is a coordinate target, not a missing feature. +- **A stale index.** Indices belong to the snapshot that produced them and expire + after 90s. A drift refusal naming a changed element is the system catching a + mis-click for you — re-snapshot and locate by LABEL, not by the old number. + +## Things that will bite you if you do not know them + +- **Electron apps are slow on the first snapshot.** Slack, VS Code, Obsidian and + KiroCrew's own desktop app need an accessibility opt-in that takes ~2 seconds to + take effect. The first `computer_get_state` on one of them looks like a hang and + is not. Wait for it; do not fire a second call. +- **A password field can look ordinary.** It renders as `` and its value + is never shown to you. A window containing one gets **no screenshot at all** — + that is deliberate, not a bug. +- **Screenshots and trees contain real user data** — file paths, window titles, + volume names, open document names. Do not echo more of a tree back to the user + than the answer needs, and never paste one into an external system. +- **Every action is real and mostly irreversible.** A click can send an email or + change a setting. Read the label before you press it, and when the outcome is + consequential say what you are about to click before you do it. +- **`element_index` is not a list position.** It is the number printed in the + outline. Use it exactly as shown. +- **Empty containers are elided** from the outline, so numbers are not always + contiguous. That is expected. +- **The user may be watching, and you cannot see what they see.** Two optional + views exist for them, not for you: a live panel that mirrors the screenshots you + take, and a "Cursor Motion" overlay that draws a moving cursor on their real + desktop along the path your next click will take. Neither is a tool and neither + changes what you get back. The overlay is deliberately **invisible to + screenshots**, including yours — so if a user says "I can see the cursor moving" + and your screenshot shows no cursor, both are correct and nothing is wrong. Do + not go looking for a cursor in a screenshot, and do not describe the overlay as + proof an action landed; the refreshed tree is the evidence. + +## Worked example + +> "Rename the top file in my Documents Finder window to notes-2026.md" + +``` +computer_get_state(app="Finder", screenshot=True) # screenshot=True opens the live view + → 0 window "Documents" + 12 row "draft.md" + ... +computer_click(app="Finder", element_index=12) # select the row +computer_press_key(app="Finder", key="return") # Finder's rename shortcut +computer_get_state(app="Finder", screenshot=True) # rename mode = a visible change + → 13 textfield "draft.md" +computer_set_value(app="Finder", element_index=13, value="notes-2026.md") +computer_press_key(app="Finder", key="return") # commit +computer_get_state(app="Finder", screenshot=True) # show the user the renamed file +computer_end_turn() +``` + +Two things to copy from this: + +- **The re-snapshot after the keypress is mandatory**, not stylistic: entering rename + mode changed the tree, so the old index for the row is no longer the index of the + field. Never carry an index across a UI change. +- **Three screenshots, not seven.** One to open the live view, one when the row turned + into a field, one to show the result. The `click`, `set_value` and the first + `press_key` produced no separate frame — they are steps toward one visible change, + and their refreshed trees already told me what I needed. diff --git a/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/diff_signals.py b/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/diff_signals.py index 77f72edcc60..86d674417f4 100755 --- a/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/diff_signals.py +++ b/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/diff_signals.py @@ -13,17 +13,17 @@ import sys SIGNALS = [ - (r"(^|/)(package\.json|requirements.*\.txt|Cargo\.toml|go\.mod|pom\.xml|" - r"build\.gradle|setup\.(py|cfg)|pyproject\.toml)", - "dependency/manifest changed - call out added/removed deps"), - (r"(^|/)(package-lock\.json|yarn\.lock|Cargo\.lock|poetry\.lock|go\.sum)", - "lockfile changed"), + ( + r"(^|/)(package\.json|requirements.*\.txt|Cargo\.toml|go\.mod|pom\.xml|" + r"build\.gradle|setup\.(py|cfg)|pyproject\.toml)", + "dependency/manifest changed - call out added/removed deps", + ), + (r"(^|/)(package-lock\.json|yarn\.lock|Cargo\.lock|poetry\.lock|go\.sum)", "lockfile changed"), (r"(migrations?/|/migrate)", "database/migration change"), (r"(^|/)\.github/workflows/", "CI workflow changed"), (r"(?m)^D\t", "files DELETED - call out removals"), (r"(?m)^R[0-9]*\t", "files RENAMED/moved"), - (r"(Dockerfile|\.tf$|\.ya?ml$|\.toml$|\.ini$|(^|/)config)", - "config/infra file changed"), + (r"(Dockerfile|\.tf$|\.ya?ml$|\.toml$|\.ini$|(^|/)config)", "config/infra file changed"), ] @@ -46,9 +46,10 @@ def main(argv): base = argv[1] if len(argv) > 1 else "" if not base: - sym = run(["git", "symbolic-ref", "--quiet", "--short", - "refs/remotes/origin/HEAD"])[1].strip() - base = sym[len("origin/"):] if sym.startswith("origin/") else "" + sym = run(["git", "symbolic-ref", "--quiet", "--short", "refs/remotes/origin/HEAD"])[ + 1 + ].strip() + base = sym[len("origin/") :] if sym.startswith("origin/") else "" if not base: base = "main" diff --git a/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/enable_automerge.py b/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/enable_automerge.py index 0efbf88ea1c..2fbce998799 100755 --- a/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/enable_automerge.py +++ b/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/enable_automerge.py @@ -30,8 +30,7 @@ def run(args): try: - p = subprocess.run(args, capture_output=True, text=True, - encoding="utf-8", errors="replace") + p = subprocess.run(args, capture_output=True, text=True, encoding="utf-8", errors="replace") return p.returncode, p.stdout.strip(), p.stderr.strip() except OSError as exc: return 127, "", "{}: {}".format(args[0], exc) @@ -52,8 +51,10 @@ def main(argv): elif arg.startswith("#") and arg[1:].isdigit(): pr = arg[1:] else: - err("ERROR: unrecognized argument '{}' (expected a PR number or one " - "of {}).".format(arg, "|".join(VALID_METHODS))) + err( + "ERROR: unrecognized argument '{}' (expected a PR number or one " + "of {}).".format(arg, "|".join(VALID_METHODS)) + ) return 2 if run(["gh", "--version"])[0] != 0: @@ -78,24 +79,29 @@ def main(argv): except ValueError: m = None if m: - print("[automerge] PR #{} already has auto-merge enabled " - "(method={}).".format(pr, m.lower())) + print( + "[automerge] PR #{} already has auto-merge enabled " + "(method={}).".format(pr, m.lower()) + ) return 0 rc, out, e = run(["gh", "pr", "merge", pr, "--auto", "--" + method]) if rc == 0: - print("[automerge] enabled auto-merge (--{}) on PR #{} - GitHub will " - "merge it once the repo's required reviews + checks are met." - .format(method, pr)) + print( + "[automerge] enabled auto-merge (--{}) on PR #{} - GitHub will " + "merge it once the repo's required reviews + checks are met.".format(method, pr) + ) return 0 err("[automerge] could not enable auto-merge on PR #{}:".format(pr)) for line in (out, e): if line: err(" " + line) - err("[automerge] common causes: 'Allow auto-merge' disabled on the repo, no " + err( + "[automerge] common causes: 'Allow auto-merge' disabled on the repo, no " "branch rule to gate it, the method is not permitted, or the PR is " - "closed/merged.") + "closed/merged." + ) return 20 diff --git a/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/preflight.py b/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/preflight.py index 5f1089ac85f..41bbe7a6a2f 100755 --- a/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/preflight.py +++ b/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/preflight.py @@ -58,23 +58,23 @@ def main(): # Base branch: prefer an existing PR's base, else origin/HEAD, else "main". base = pr_base if not base: - sym = run(["git", "symbolic-ref", "--quiet", "--short", - "refs/remotes/origin/HEAD"])[1] + sym = run(["git", "symbolic-ref", "--quiet", "--short", "refs/remotes/origin/HEAD"])[1] if sym.startswith("origin/"): - base = sym[len("origin/"):] + base = sym[len("origin/") :] else: base = sym if not base: base = "main" - on_protected = (cur == base) + on_protected = cur == base dirty = bool(run(["git", "status", "--porcelain"])[1]) # Divergence vs base (non-destructive fetch). behind = ahead = "?" run(["git", "fetch", "--quiet", "origin", base]) - rc, out, _ = run(["git", "rev-list", "--left-right", "--count", - "origin/{}...HEAD".format(base)]) + rc, out, _ = run( + ["git", "rev-list", "--left-right", "--count", "origin/{}...HEAD".format(base)] + ) if rc == 0 and len(out.split()) == 2: behind, ahead = out.split() @@ -85,17 +85,20 @@ def main(): print("working tree: " + ("dirty" if dirty else "clean")) print("vs origin/{}: behind={} ahead={}".format(base, behind, ahead)) print("gh authed: " + ("yes" if gh_ok else "no")) - print("existing PR: " + (pr_num or "none") - + ((" (" + pr_url + ")") if pr_url else "")) + print("existing PR: " + (pr_num or "none") + ((" (" + pr_url + ")") if pr_url else "")) blocked = False if cur == "HEAD": - print("BLOCKER: detached HEAD (no branch checked out) - switch to a " - "feature branch first: git switch -c /") + print( + "BLOCKER: detached HEAD (no branch checked out) - switch to a " + "feature branch first: git switch -c /" + ) blocked = True elif on_protected: - print("BLOCKER: on the integration branch '{}' - create a feature branch " - "first: git switch -c /".format(cur)) + print( + "BLOCKER: on the integration branch '{}' - create a feature branch " + "first: git switch -c /".format(cur) + ) blocked = True if not gh_ok: print("BLOCKER: gh not authenticated - run: gh auth login") diff --git a/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/resolve_profile.py b/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/resolve_profile.py index 5047f70ca3b..1084d75ed63 100644 --- a/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/resolve_profile.py +++ b/src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/scripts/resolve_profile.py @@ -186,6 +186,7 @@ def load_bundled(name): def detect_gates(root): """Infer local gate commands from ecosystem marker files.""" + def has(rel): return os.path.exists(os.path.join(root, rel)) diff --git a/src/kiro_crew/cli.py b/src/kiro_crew/cli.py index 0ab843e515d..a5d7083a7dc 100644 --- a/src/kiro_crew/cli.py +++ b/src/kiro_crew/cli.py @@ -1411,6 +1411,11 @@ def _tunnel_opts(p: "argparse.ArgumentParser") -> None: # mcp-core (MCP server — spawned by the agent backend, not user-facing) sub.add_parser("mcp-core", help=argparse.SUPPRESS) + # mcp-computer (MCP server — spawned by the agent backend, not user-facing). + # A THIN SHIM: it forwards to the gateway over loopback, where the + # fail-closed governance gate and all accessibility work live. + sub.add_parser("mcp-computer", help=argparse.SUPPRESS) + # Builtin app MCP servers (spawned by the agent backend, not user-facing) for _bname in _BUILTIN_NAMES: sub.add_parser(f"mcp-{_bname}", help=argparse.SUPPRESS) @@ -1436,6 +1441,30 @@ def _tunnel_opts(p: "argparse.ArgumentParser") -> None: help="browse sub-command and its arguments", ) + # computer — computer-use (desktop automation) diagnostics. READ-ONLY: there + # is deliberately no CLI verb that reads a window's contents or drives an + # app, because those are LLM-facing capabilities and the MCP-first rule puts + # them in the ``kirocrew-computer`` MCP server instead. + computer_parser = sub.add_parser( + "computer", + help="Computer-use (desktop automation) diagnostics", + epilog=""" +Examples: + kirocrew computer doctor # Support + permission report + kirocrew computer doctor --json # The same report as JSON + kirocrew computer apps # Apps with an on-screen window + +Computer use is OFF by default and is enabled only from the dashboard +(Settings -> Computer Use). An agent cannot enable it. +""", + formatter_class=_fmt, + ) + computer_parser.add_argument( + "computer_args", + nargs=argparse.REMAINDER, + help="computer sub-command and its arguments", + ) + # learn learn_parser = sub.add_parser( "learn", @@ -1885,11 +1914,23 @@ def _tunnel_opts(p: "argparse.ArgumentParser") -> None: from kiro_crew.mcp_core import run_mcp_core_server run_mcp_core_server() + elif args.command == "mcp-computer": + from kiro_crew.mcp_computer import run_mcp_server as run_mcp_computer_server + + run_mcp_computer_server() elif args.command.startswith("mcp-") and args.command[4:] in _BUILTIN_NAMES: _mod = importlib.import_module(f"kiro_crew.apps.builtins.{args.command[4:]}.mcp_server") _mod.run_mcp_server() elif args.command == "browse": run_browse(getattr(args, "browse_args", [])) + elif args.command == "computer": + # Deferred import: ``computer_use.cli`` reaches the driver seam, and the + # macOS driver loads native frameworks on first use. Keeping it out of + # cli.py's module imports means every OTHER command — and the whole CI + # fleet — pays nothing for it. + from kiro_crew.computer_use.cli import run_computer + + run_computer(getattr(args, "computer_args", [])) elif args.command == "eval": asyncio.run(_run_eval(args)) elif args.command == "security": diff --git a/src/kiro_crew/cli_doctor.py b/src/kiro_crew/cli_doctor.py index 07e3337a89e..68f7e367807 100644 --- a/src/kiro_crew/cli_doctor.py +++ b/src/kiro_crew/cli_doctor.py @@ -51,6 +51,7 @@ current_context, safe_context_call, ) +from kiro_crew.platform.governance import CU_MCP_SERVER from kiro_crew.transcribe import _find_whisper, ensure_ffmpeg_in_path _MIN_NODE_VERSION = 16 @@ -67,16 +68,30 @@ def _os_fix_hint(mac: str, linux: str) -> str: # optional seam rather than as a user-facing backend. _CLAUDE_ACP_BIN = "claude-agent-acp" +# Managed servers doctor must NEVER add to ``allowedTools``. +# +# ``allowedTools`` is kiro-cli's blanket auto-approve list, and an auto-approved +# MCP tool is approved LOCALLY by kiro-cli: it emits no permission request and +# therefore NEVER reaches ``hooks.on_tool_call`` — the PreToolUse plane that +# carries the always-on deny floor, the sensitive-path check and the governance +# ceiling. ``agent.py``'s managed spec deliberately omits ``autoApprove`` for +# exactly this reason (a tool that can click and type into an +# already-authenticated application must stay behind a prompt), and a diagnostic +# command must not silently undo that. Doctor still repairs the ``tools`` entry, +# which only makes the server's tools *reachable*, never pre-approved. +_NO_BLANKET_ALLOW_MCPS = frozenset({CU_MCP_SERVER}) + def _doctor_mcp_tools(agent_path: Path, issues: list[str]) -> None: """Render the `MCP Tools` section of `kirocrew doctor`. Two passes scoped to the managed servers (`kirocrew-core`, - `kirocrew-cron`): + `kirocrew-cron`, `kirocrew-computer`): 1. Static sanity check of the agent config: each server must be present - in ``mcpServers``, ``tools`` and ``allowedTools``. Missing ``tools`` - / ``allowedTools`` entries are auto-appended and the file is + in ``mcpServers`` and ``tools``. Missing ``tools`` entries — and + ``allowedTools`` entries for every server outside + :data:`_NO_BLANKET_ALLOW_MCPS` — are auto-appended and the file is rewritten atomically. A missing ``mcpServers`` entry cannot be auto-added because the command path is install-specific. 2. Live handshake probe via :func:`mcp_discovery.probe_server`. Reports @@ -105,7 +120,10 @@ def _doctor_mcp_tools(agent_path: Path, issues: list[str]) -> None: if ref not in tools: tools.append(ref) config_changed = True - if ref not in allowed: + # Computer use is never blanket-allowed here: see _NO_BLANKET_ALLOW_MCPS. + # A pre-existing user-made grant is left alone (doctor never REMOVES a + # decision the user owns); doctor simply never mints one. + if ref not in allowed and name not in _NO_BLANKET_ALLOW_MCPS: allowed.append(ref) config_changed = True diff --git a/src/kiro_crew/computer_use/__init__.py b/src/kiro_crew/computer_use/__init__.py new file mode 100644 index 00000000000..3e5c68a337a --- /dev/null +++ b/src/kiro_crew/computer_use/__init__.py @@ -0,0 +1,72 @@ +"""KiroCrew computer use — read and drive desktop app windows via accessibility. + +Public surface of the package. Importing it is **side-effect free**: no native +framework is loaded, no ``CDLL`` runs, no file is read, and no platform branch is +taken until :func:`get_shared_backend` is actually called. That is what lets the +Linux and Windows CI shards import the whole package and exercise its +platform-free logic without a driver. + +Architecture in one paragraph: the ``kirocrew-computer`` MCP sidecar dispatches +tool calls into one synchronous chokepoint, which checks the keystone primary +enable, then governance, then the target policy, then the snapshot index's +freshness and fingerprint, and only then calls a +:class:`~kiro_crew.computer_use.backend.ComputerUseBackend`. The accessibility +tree is the primary channel; a screenshot is compressed, persisted to an +owner-only temp dir, and relayed only as a path. + +See ``docs/system-specs/modules/computer-use.md``. +""" + +from __future__ import annotations + +from kiro_crew.computer_use.backend import ( + ComputerUseBackend, + UnsupportedBackend, + get_shared_backend, + platform_id_for_current_os, + register_computer_use_backend, + reset_shared_backend, + select_default_backend, +) +from kiro_crew.computer_use.index import SnapshotIndex, get_shared_index, reset_shared_index +from kiro_crew.computer_use.types import ( + AppRef, + BackendStatus, + ComputerUseDenied, + ComputerUseError, + ComputerUseUnsupported, + DriverResult, + ElementRec, + KeyParseError, + PermissionProbe, + PolicyConfig, + Snapshot, + SnapshotRequest, + StaleIndex, +) + +__all__ = [ + "AppRef", + "BackendStatus", + "ComputerUseBackend", + "ComputerUseDenied", + "ComputerUseError", + "ComputerUseUnsupported", + "DriverResult", + "ElementRec", + "KeyParseError", + "PermissionProbe", + "PolicyConfig", + "Snapshot", + "SnapshotIndex", + "SnapshotRequest", + "StaleIndex", + "UnsupportedBackend", + "get_shared_backend", + "get_shared_index", + "platform_id_for_current_os", + "register_computer_use_backend", + "reset_shared_backend", + "reset_shared_index", + "select_default_backend", +] diff --git a/src/kiro_crew/computer_use/apps_macos.py b/src/kiro_crew/computer_use/apps_macos.py new file mode 100644 index 00000000000..34085fd76e7 --- /dev/null +++ b/src/kiro_crew/computer_use/apps_macos.py @@ -0,0 +1,486 @@ +"""Application discovery and OS-resolved identity, from the window list ONLY. + +Two jobs, and the second is security-critical: + +1. :func:`list_apps` / :func:`resolve_app` answer "which applications have a + visible window, and which pid owns the one the agent named". +2. :func:`resolve_identity` produces the **OS-resolved** bundle id and display + name that ``computer_use/gate.py`` queries governance on. The gate never + queries the agent-supplied ``app`` string: if it did, an allow-listed + ``com.apple.finder`` could be satisfied by a model claiming "finder" while the + automation actually drove something else. Everything here answers "what did we + really touch", not "what were we asked for". + +**Never a process-name search.** Not ``pgrep``, not ``ps``, not a name scan of +any kind — the string does not appear in this module and a test asserts that. +The reason is a reproduced failure, not a style preference: ``pgrep -n "Google +Chrome"`` returned 47492, a short-lived helper process that answered +``kAXErrorCannotComplete`` to every accessibility read and then exited, while the +real browser was 637. Slack's helper 1614 likewise shadowed the real 942. The pid +that owns a visible, layer-0 window is the one whose accessibility tree is +populated, so the CoreGraphics window list is the only source of truth. + +Bundle-id resolution is a **live finding that contradicts the obvious design.** +Neither source you would reach for first works: + +* the window-list dictionary carries **no bundle-id key at all** (verified: the + dict is exactly ``{Alpha, Bounds, IsOnscreen, Layer, MemoryUsage, Name, Number, + OwnerName, OwnerPID, SharingState, StoreType}``); +* ``AXBundleIdentifier`` on the application element returns + ``kAXErrorAttributeUnsupported`` (-25205) for **every** app on this macOS — + Chrome, Slack, Notes, Obsidian, Outlook, System Settings, all of them. It is + not an Electron quirk and not a permissions artifact. + +So the bundle id comes from ``proc_pidpath`` plus the enclosing bundle's +``Info.plist`` — one syscall and one small plist read, measured under 1ms per pid, +verified correct for all eight apps on screen (``com.google.Chrome``, +``com.tinyspeck.slackmacgap``, ``md.obsidian``, ``com.apple.Notes``, +``com.apple.systempreferences``, …). An unbundled binary yields ``""`` and the +process name carries the identity instead; the gate treats a fully unresolved +identity as a DENY, which is the correct posture for a target we cannot name. +""" + +from __future__ import annotations + +import logging +import os +import plistlib +import threading +import time +from dataclasses import dataclass + +from kiro_crew.computer_use import macos_ffi +from kiro_crew.computer_use.policy import title_is_denied as _denied_title +from kiro_crew.computer_use.types import ( + ERR_APP_NOT_FOUND, + TOOL_LIST_APPS, + AppRef, + ComputerUseError, +) + +logger = logging.getLogger(__name__) + +# Info.plist keys, in the order :func:`_bundle_metadata` prefers them for a +# display name. ``CFBundleName`` before ``CFBundleDisplayName`` because the former +# is the short canonical name ("Chrome") users and governance patterns type, while +# the latter is often a marketing string. +PLIST_BUNDLE_ID_KEY = "CFBundleIdentifier" +PLIST_BUNDLE_NAME_KEYS: tuple[str, ...] = ("CFBundleName", "CFBundleDisplayName") +BUNDLE_SUFFIX = ".app" +BUNDLE_INFO_RELPATH = os.path.join("Contents", "Info.plist") + +# Largest Info.plist we will parse. A bundle's Info.plist is a few KB; anything +# far larger is either not what we think it is or a resource-exhaustion attempt +# through a hand-crafted bundle, and the identity is not worth an unbounded read. +MAX_INFO_PLIST_BYTES = 512 * 1024 + +# Identity cache. A pid's bundle identity cannot change while the process lives, +# so caching it turns a per-action plist read into a dict hit. The TTL bounds pid +# reuse: a recycled pid belonging to a different program must not inherit the +# previous occupant's identity, since that identity is what governance is queried +# on. 60s is far shorter than any realistic reuse window on macOS (pids advance +# monotonically through a large space) while still covering a whole turn. +IDENTITY_CACHE_TTL_SECS = 60.0 +MAX_CACHED_IDENTITIES = 64 + + +@dataclass(frozen=True) +class AppIdentity: + """The OS-resolved identity of one process. + + ``bundle_id`` is the stable OS identity and ``display_name`` is the + human-facing one. ``policy.check_app`` matches them SEPARATELY (never + pipe-joined against one pattern), because an app's display name is + attacker-chooseable and must not be able to satisfy a bundle-id rule. + + ``executable`` is kept for diagnostics: when both identifiers come back empty + it is the only thing that can tell an operator what the sidecar was looking + at. + """ + + bundle_id: str = "" + display_name: str = "" + executable: str = "" + + @property + def resolved(self) -> bool: + """True when at least one governable identifier is known. + + The gate refuses a call with no identity at all: under a deny-mode + ``apps`` ruleset an empty item matches no pattern and would therefore + PERMIT, so governance cannot bound a target it cannot name. + """ + return bool(self.bundle_id or self.display_name) + + +_identity_cache: dict[int, tuple[float, AppIdentity]] = {} +_identity_lock = threading.Lock() + + +def list_apps() -> tuple[AppRef, ...]: + """Applications with at least one on-screen, layer-0 window. + + One :class:`AppRef` per application (not per window), keyed by pid: an app + with six windows is one entry. + + **The window we keep is the frontmost one that has a TITLE**, not simply the + first layer-0 entry. Window-list order is CoreGraphics' front-to-back z-order, + so "first" is usually right — but verified live on Notes, the frontmost layer-0 + window was an untitled ``AXDialog`` exposing 3 accessibility nodes, sitting in + front of the real 103-node document window. Taking it verbatim would make + ``computer_get_state`` on Notes return a near-empty tree while the app clearly + had content, with nothing in the result explaining why. An untitled window is + still kept when the app has no titled one at all (a splash screen, a bare + alert) — the app is genuinely showing that. + + Non-zero layers are dropped: menu bars, the Dock, tooltips, notification + banners and window shadows all live there, they are not addressable + application windows, and their owning pid is frequently a system agent whose + tree is empty. + """ + seen: dict[int, AppRef] = {} + for info in macos_ffi.window_list(): + if info.layer != macos_ffi.CG_WINDOW_LAYER_NORMAL: + continue + if info.pid <= 0 or not info.owner_name: + continue + existing = seen.get(info.pid) + if existing is not None and _denied_title(existing.window_title): + # Already holding a window whose TITLE trips the denylist. Never + # replace it: input is delivered per-PID (``CGEventPostToPid``), so if + # ANY window of this process is our own dashboard the whole process must + # refuse. Keeping the innocuous first window would let a second Chrome + # window hosting KiroCrew slip past the title rule entirely. + continue + if existing is not None and not _denied_title(info.title): + if existing.window_title or not info.title: + # Already have a window for this app, and either it is titled (good + # enough) or this candidate is not an improvement. + continue + identity = resolve_identity(info.pid) + seen[info.pid] = AppRef( + # The window list's owner name is the process name; the bundle's + # CFBundleName is usually the nicer one ("Chrome" vs "Google + # Chrome"), but the process name is what the operator sees in + # Activity Monitor and what the denylist's name substrings match, so + # it stays authoritative for ``name``. + name=info.owner_name, + pid=info.pid, + bundle_id=identity.bundle_id, + window_id=info.window_id, + window_title=info.title, + ) + return tuple(seen.values()) + + +def resolve_app(query: str) -> AppRef: + """Resolve *query* (process name, bundle id, or a fragment) to one app. + + Raises :class:`ComputerUseError` when nothing matches, naming + ``computer_list_apps`` so the model's next move is obvious. + + Match order, most specific first, so a precise query is never captured by a + loose one: + + 1. exact bundle id; + 2. exact process name (case-insensitive); + 3. bundle-id substring; + 4. process-name substring. + + Within a tier the frontmost window wins (window-list order). The tiers matter + for a real ambiguity: a query of ``"notes"`` should reach ``com.apple.Notes`` + rather than an app whose window title happens to contain the word. + """ + needle = (query or "").strip().lower() + if not needle: + raise ComputerUseError(ERR_APP_NOT_FOUND.format(query=query, tool=TOOL_LIST_APPS)) + apps = list_apps() + + for app in apps: + if app.bundle_id and app.bundle_id.lower() == needle: + return app + for app in apps: + if app.name and app.name.lower() == needle: + return app + for app in apps: + if app.bundle_id and needle in app.bundle_id.lower(): + return app + for app in apps: + if app.name and needle in app.name.lower(): + return app + raise ComputerUseError(ERR_APP_NOT_FOUND.format(query=query, tool=TOOL_LIST_APPS)) + + +def resolve_identity(pid: int) -> AppIdentity: + """The OS-resolved identity of *pid*. Never raises. + + This is the function ``gate.require_computer_use`` is fed from, so it must + answer "what is this process, according to the OS" and nothing else — no + agent-supplied string reaches it. + + Resolution path and why it is not the obvious one: + + * ``AXBundleIdentifier`` is **not** consulted. Verified to return + ``kAXErrorAttributeUnsupported`` for every application on this macOS, so a + code path built on it would resolve nothing while looking correct. + * The window-list dictionary carries no bundle-id key either. + * So: ``proc_pidpath(pid)`` gives the absolute executable path, the nearest + enclosing ``*.app`` directory gives the bundle, and its ``Info.plist`` + gives ``CFBundleIdentifier`` + ``CFBundleName``. Verified correct for every + app on screen, sub-millisecond per pid. + + Returns an empty-but-``resolved``-False identity for an unbundled binary or a + dead pid. That is deliberately NOT patched up with a guess: the gate denies an + unidentifiable target, which is the right answer, and inventing an identity + would create a governable name the operator never approved. + """ + if pid <= 0: + return AppIdentity() + now = time.monotonic() + with _identity_lock: + cached = _identity_cache.get(pid) + if cached is not None and now - cached[0] <= IDENTITY_CACHE_TTL_SECS: + return cached[1] + + identity = _resolve_identity_uncached(pid) + + with _identity_lock: + # Evict expired entries opportunistically, then bound the map. Only a + # handful of apps are in play per turn; the cap exists so a long-lived + # sidecar cannot accumulate an entry per pid it ever saw. + for key, (stamp, _) in list(_identity_cache.items()): + if now - stamp > IDENTITY_CACHE_TTL_SECS: + _identity_cache.pop(key, None) + while len(_identity_cache) >= MAX_CACHED_IDENTITIES: + _identity_cache.pop(next(iter(_identity_cache)), None) + _identity_cache[pid] = (now, identity) + return identity + + +def reset_identity_cache() -> None: + """Drop the identity cache (driver close, tests).""" + with _identity_lock: + _identity_cache.clear() + + +def identity_for(app: AppRef) -> AppIdentity: + """Identity for an already-resolved :class:`AppRef`. + + Re-resolves from the pid rather than trusting the ``AppRef``'s own fields: + the ref may have been built from a cached snapshot, and the gate must be fed + a freshly observed identity. Falls back to the ref's process name for + ``display_name`` so an unbundled app is still governable by name. + """ + identity = resolve_identity(app.pid) + if identity.display_name or not app.name: + return identity + return AppIdentity( + bundle_id=identity.bundle_id, + display_name=app.name, + executable=identity.executable, + ) + + +def _resolve_identity_uncached(pid: int) -> AppIdentity: + """Do the real resolution work for one pid. Never raises.""" + try: + executable = macos_ffi.executable_path(pid) + except Exception: + # A dead pid, a sandbox refusal, or an FFI problem: identity unknown is a + # valid answer here and the gate handles it (by denying). + logger.debug("proc_pidpath failed for pid %s", pid, exc_info=True) + return AppIdentity() + if not executable: + return AppIdentity() + bundle_id, bundle_name = _bundle_metadata(executable) + return AppIdentity(bundle_id=bundle_id, display_name=bundle_name, executable=executable) + + +def _bundle_metadata(executable: str) -> tuple[str, str]: + """``(bundle_id, display_name)`` from the bundle enclosing *executable*. + + Walks UP from the executable looking for the nearest ``*.app`` directory, + because the binary always sits at ``Foo.app/Contents/MacOS/Foo`` and a helper + can be nested deeper still. Returns ``("", "")`` for a plain binary. + """ + bundle = _enclosing_bundle(executable) + if not bundle: + return "", "" + info = _read_info_plist(os.path.join(bundle, BUNDLE_INFO_RELPATH)) + if not info: + return "", "" + bundle_id = info.get(PLIST_BUNDLE_ID_KEY) + bundle_id = bundle_id.strip() if isinstance(bundle_id, str) else "" + display = "" + for key in PLIST_BUNDLE_NAME_KEYS: + value = info.get(key) + if isinstance(value, str) and value.strip(): + display = value.strip() + break + return bundle_id, display + + +def _enclosing_bundle(executable: str) -> str: + """The nearest ancestor directory ending in ``.app``, or ``""``. + + Bounded by the path's own depth: the loop terminates when ``dirname`` stops + changing, so a malformed or relative path cannot spin. + """ + current = os.path.abspath(executable) + while True: + parent = os.path.dirname(current) + if parent == current: + return "" + current = parent + if current.endswith(BUNDLE_SUFFIX): + return current + + +def _read_info_plist(path: str) -> dict: + """Parse a bundle ``Info.plist``, or return ``{}``. Never raises. + + Size-capped and fully defensive. This reads a file path derived from a + process the AGENT chose to target, so it must tolerate a hand-crafted bundle + with a malformed, enormous, or hostile plist — and the consequence of failing + is only "identity unknown", which the gate already handles by denying. + + ``plistlib`` is stdlib and parses both the binary and XML formats macOS uses. + + Read through ``hooks.safe_read_prefix``, which is the repo's hardened path for + any read of an agent-influenced location: it canonicalizes with ``realpath``, + re-checks the RESOLVED target against ``is_sensitive_path``, and opens with + ``O_NOFOLLOW``. + + All three matter here. The path comes from the OS (the target process's + executable, walked up to its ``.app`` ancestor) rather than from the agent, but + the agent DOES choose which process to target — and therefore can arrange the + bundle. So a bundle planted under a protected directory (``~/.ssh/evil.app``) + must not have its ``Info.plist`` read, and — the part an earlier revision got + wrong — checking the path and then opening it separately is check-then-open: a + final-component symlink swapped in between would read a protected file on a path + that never consulted the floor. ``O_NOFOLLOW`` on the canonical path is what + closes that window. + + The floor is about the file being opened, not about who computed the string, and + the cost of honouring it is only "identity unknown" for a bundle in a place no + application belongs — which the gate already handles by denying. + """ + try: + from kiro_crew.hooks import safe_read_prefix + + # Through ``safe_read_prefix``, not a bare ``open`` — GPT 5.6 BLOCKING, and + # correct. The hand-rolled version checked ``is_sensitive_path`` and then + # opened the path separately, which is check-then-open: the agent chooses + # which process to target, so it can also arrange the bundle, and a + # final-component symlink swapped between the check and the open would have + # read a protected file's bytes on a path that never touched the hardened + # gate. The helper canonicalizes with ``realpath``, re-checks the RESOLVED + # target, and opens with ``O_NOFOLLOW`` — which is the TOCTOU defence this + # code was missing, and the repo's stated requirement for any read of an + # agent-influenced path. + # + # Reading ``MAX_INFO_PLIST_BYTES + 1`` rather than the cap: the extra byte is + # what distinguishes "exactly at the limit" from "larger than the limit" + # without a second ``getsize`` call, and the size check has to be on the bytes + # actually READ rather than on a stat of a path that may no longer be the same + # file — the same race, one step further along. + raw = safe_read_prefix(path, MAX_INFO_PLIST_BYTES + 1) + if raw is None: + logger.debug("refusing to read an Info.plist on the sensitive-path floor") + return {} + if len(raw) > MAX_INFO_PLIST_BYTES: + logger.debug("Info.plist too large to parse: %s", path) + return {} + parsed = plistlib.loads(raw) + except Exception: + logger.debug("Info.plist unreadable: %s", path, exc_info=True) + return {} + return parsed if isinstance(parsed, dict) else {} + + +def window_bounds(window_id: int, pid: int) -> "tuple[float, float, float, float] | None": + """Screen rect of *window_id*, or ``None`` if it is gone or not owned by *pid*. + + ``(left, top, width, height)`` in the package's top-left convention. Needed by + ``sky_click``, which routes by a WINDOW-LOCAL point and therefore has to convert + the caller's screen coordinate — the window server ignores the screen point on + that path and reads the local one. + + The *pid* is verified rather than trusted: window ids are recycled, so a stale + id can name a different app's window, and sky_click's whole purpose is clicking + something the operator cannot see (a mis-aimed click there is invisible). + Returns ``None`` on any error so the caller refuses rather than guessing a rect. + """ + try: + for info in macos_ffi.window_list(): + if info.window_id != int(window_id): + continue + if info.pid != int(pid) or info.bounds is None: + return None + return info.bounds + except Exception: + logger.debug("window_bounds lookup failed for %s", window_id, exc_info=True) + return None + + +def pid_owns_point(pid: int, x: float, y: float) -> bool: + """Does *pid* own the topmost on-screen window containing ``(x, y)``? + + THE confinement check for the real-pointer click path, and the reason it has + to exist: ``CGWarpMouseCursorPosition`` + a global ``CGEventPost`` deliver to + whatever is under that pixel, with no pid in the call. Every other input path + is app-scoped (``CGEventPostToPid``), so the resolved app identity IS the + authorization boundary — ``policy.check_app`` ran the denylist against THAT + app. Without this check, naming an allowed app and passing coordinates over a + denied one (KiroCrew's own window) would sail through: the policy check passes + on app A while the click lands on app B. + + Topmost, not merely "inside": the window list is front-to-back z-order, so the + FIRST layer-0 entry whose rect contains the point is the window that would + actually receive the click. Testing "any window of *pid* contains the point" + would wrongly permit a click that visually lands on a denied app overlapping + the allowed one. + + Fails CLOSED (``False``) when the point belongs to nobody, when the owning + window carries no readable bounds, or on any error — refusing a legitimate + click costs the model one clear refusal, while permitting a mis-aimed one is + an irreversible action in an app the operator never authorized. + """ + try: + for info in macos_ffi.window_list(): + bounds = info.bounds + if bounds is None: + # Cannot place this window, so cannot prove it does NOT cover the + # point. Refuse rather than look past it. + return False + left, top, width, height = bounds + if not (left <= x < left + width and top <= y < top + height): + continue + # FIRST containing window in z-order wins, whatever its layer. A + # non-normal window is NOT skipped here, which is the opposite of what + # ``list_apps`` does and deliberately so: this function answers "what + # would a physical click at this pixel hit?", and a notification banner, + # a menu-bar extra or an open menu sitting above the authorized app + # really would receive that click. Skipping those layers let the app + # UNDERNEATH grant permission for a click the operator's own overlay was + # about to swallow — the very confinement this function exists to + # provide. ``list_apps`` skips them for the unrelated reason that they + # are not addressable TARGETS. + return info.pid == pid and info.layer == macos_ffi.CG_WINDOW_LAYER_NORMAL + return False + except Exception: + logger.debug("point ownership check failed; refusing", exc_info=True) + return False + + +__all__ = [ + "AppIdentity", + "IDENTITY_CACHE_TTL_SECS", + "MAX_INFO_PLIST_BYTES", + "identity_for", + "list_apps", + "pid_owns_point", + "reset_identity_cache", + "resolve_app", + "resolve_identity", +] diff --git a/src/kiro_crew/computer_use/backend.py b/src/kiro_crew/computer_use/backend.py new file mode 100644 index 00000000000..a5eea8e8ccb --- /dev/null +++ b/src/kiro_crew/computer_use/backend.py @@ -0,0 +1,396 @@ +"""The computer-use backend seam: one ABC, one registry, one platform branch. + +``ComputerUseBackend`` is what varies by OS; everything above it (policy, +rendering, the index lifecycle, the MCP dispatch) is platform-free. The seam is a +plain ``abc.ABC`` with a runtime-swappable factory — the shape +``embeddings.register_embedding_backend`` already uses — rather than a +``PlatformContext`` extension point, because CPP is the *edition* seam +(standalone vs companion) while computer use varies by *platform*, and because a +registry stays swappable inside a single pytest process. + +Two structural guarantees: + +* **One platform branch.** :func:`select_default_backend` is the only place in + the whole package that asks which OS this is, and it asks + ``platform_compat.IS_MACOS`` / ``IS_WINDOWS`` / ``IS_LINUX`` rather than + reading ``sys.platform`` itself. +* **No exception crosses the seam.** Every method returns a + :class:`DriverResult`; drivers convert their own failures into + ``ok=False``. A ``ComputerUseError`` escaping into the MCP stdio loop's worker + thread would take the tool call down, and an unhandled ctypes fault would take + the sidecar with it. + +Nothing in this module imports ctypes, and the native drivers are imported +lazily inside :func:`select_default_backend` — so importing this package on a +Linux CI runner loads no native library at all. +""" + +from __future__ import annotations + +import abc +import logging +import threading +from typing import Callable + +from kiro_crew import platform_compat +from kiro_crew.computer_use import index +from kiro_crew.computer_use.types import ( + PERMISSION_UNSUPPORTED, + PLATFORM_LINUX, + PLATFORM_MACOS, + PLATFORM_UNSUPPORTED, + PLATFORM_WINDOWS, + REFUSAL_UNSUPPORTED, + AppRef, + BackendStatus, + ClickRequest, + DragRequest, + DriverResult, + ElementRec, + PermissionProbe, + Snapshot, + SnapshotRequest, +) + +logger = logging.getLogger(__name__) + +# Reasons the non-macOS backends report. Concrete rather than "not supported": +# a user on Windows should learn what is missing, and a maintainer should find +# the next implementation step named here. +WINDOWS_REASON = ( + "the Windows UI Automation driver is not implemented yet; computer use is " + "macOS-only in this release" +) +LINUX_REASON = ( + "the Linux AT-SPI driver is not implemented yet; computer use is macOS-only " + "in this release (Wayland has no unprivileged window capture)" +) +UNKNOWN_PLATFORM_REASON = "this operating system has no computer-use driver" +DRIVER_IMPORT_REASON = "the native computer-use driver could not be loaded ({detail})" + + +class ComputerUseBackend(abc.ABC): + """Abstract desktop-automation driver for one platform. + + Contract every implementation must honor: + + * **Never raise.** Convert every failure — a missing app, an accessibility + error, a permission denial — into ``DriverResult(ok=False, text=)`` + with the reason WITHOUT an ``Error: `` prefix (the dispatch layer adds it + exactly once). + * **Never move the pointer unless the request says to.** Every method is + app-scoped by default: an element click is an accessibility action and a + coordinate click/drag posts to the target process, so the operator's cursor + does not move. The ONE exception is a :class:`ClickRequest` / + :class:`DragRequest` whose ``moves_pointer`` is True (the ``global`` + method), which the model must have NAMED explicitly — ``auto`` never resolves + to it. A driver MUST NOT warp the cursor for any other method, and MUST NOT + silently upgrade a refused app-scoped click into a pointer-moving one. + * **Set ``ElementRec.secure`` from BOTH ``AXRole`` and ``AXSubrole``** (or + the platform equivalent). A password box reports an innocuous role with a + secure *subrole* and a readable value; a role-only check misses every one. + Every downstream protection — value redaction, input refusal, screenshot + suppression — keys off this flag, so a driver that gets it wrong defeats + all three at once. + * **Set ``Snapshot.captured_at`` from ``time.monotonic()``**, never wall + clock, so the TTL cannot be defeated by a clock adjustment. + * **Be thread-safe.** The MCP loop dispatches on a worker thread while the + main thread reads stdin. + """ + + @property + @abc.abstractmethod + def platform_id(self) -> str: + """Stable platform identifier (``macos``/``windows``/``linux``/``fake``).""" + + @abc.abstractmethod + def status(self) -> BackendStatus: + """Whether this backend can drive computer use here, and why not if it can't.""" + + @abc.abstractmethod + def probe_permissions(self) -> PermissionProbe: + """ADVISORY permission hints for the Settings UI — never a gate. + + macOS attributes a TCC grant to the responsible parent of the process + tree, so a probe can report ``missing`` while a full-fidelity capture + succeeds. Callers must not refuse an action based on this. + """ + + @abc.abstractmethod + def list_apps(self) -> DriverResult: + """Applications with an on-screen window, resolved from the window list.""" + + @abc.abstractmethod + def resolve_app(self, query: str) -> DriverResult: + """Resolve *query* (name or bundle id) to one :class:`AppRef`. + + MUST resolve from the on-screen window list, never from a process-name + search: a ``pgrep``-style match returns short-lived helper processes + whose accessibility trees are empty. + """ + + @abc.abstractmethod + def snapshot(self, app: AppRef, req: SnapshotRequest) -> DriverResult: + """Walk *app*'s focused window into a :class:`Snapshot`. + + Honors every budget in *req*. When ``req.want_image`` is set the + snapshot MAY carry encoded JPEG bytes; a driver that cannot capture + returns the tree alone rather than failing the call. + """ + + @abc.abstractmethod + def click( + self, + app: AppRef, + rec: "ElementRec | None", + req: ClickRequest, + ) -> DriverResult: + """Click *rec* (accessibility press) or *req*'s point (mouse event). + + *req* arrives with a CONCRETE method — ``auto`` is resolved at the dispatch + chokepoint, so a driver never re-decides it and cannot accidentally pick the + pointer-warping path. ``rec`` is ``None`` for a coordinate click, and + ``req.point`` is ``None`` for the element form; exactly one is set (validated + upstream by ``policy.check_click_target``). + + Honour ``req.button`` and ``req.count`` for a mouse-event click, and warp the + pointer ONLY when ``req.moves_pointer`` is True. + """ + + @abc.abstractmethod + def drag(self, app: AppRef, req: DragRequest) -> DriverResult: + """Drag from *req*'s start point to its end point inside *app*. + + Coordinate-only by construction: a drag's meaning IS the path between two + points, and no accessibility action expresses it. App-scoped unless + ``req.moves_pointer`` is True, in which case the two upstream permits have + already been checked. + """ + + @abc.abstractmethod + def type_text(self, app: AppRef, rec: "ElementRec | None", text: str) -> DriverResult: + """Type *text* into *rec*, or into the focused element when *rec* is None.""" + + @abc.abstractmethod + def press_key(self, app: AppRef, rec: "ElementRec | None", key: str) -> DriverResult: + """Send one key spec (``"cmd+shift+a"``) to *app*.""" + + @abc.abstractmethod + def set_value(self, app: AppRef, rec: ElementRec, value: str) -> DriverResult: + """Set *rec*'s value directly (no keystrokes).""" + + @abc.abstractmethod + def scroll(self, app: AppRef, rec: ElementRec, direction: str, pages: float) -> DriverResult: + """Scroll *rec* by *pages* in *direction*.""" + + @abc.abstractmethod + def perform_action(self, app: AppRef, rec: ElementRec, action: str) -> DriverResult: + """Perform a named accessibility action on *rec*.""" + + @abc.abstractmethod + def close(self) -> None: + """Release resources. Safe to call repeatedly.""" + + +class UnsupportedBackend(ComputerUseBackend): + """Shared base for platforms with no driver: every method refuses identically. + + Concrete on purpose — a subclass supplies only ``platform_id`` and a reason, + so a new unsupported platform is ~10 lines and CANNOT accidentally implement + half a driver. Nothing here raises: the model gets a coherent refusal naming + the platform instead of a broken capability, which is the same posture + ``dashboard/handlers/terminal.py`` takes for the Windows PTY. + """ + + def __init__(self, platform_id: str, reason: str) -> None: + self._platform_id = platform_id + self._reason = reason + + @property + def platform_id(self) -> str: + return self._platform_id + + @property + def reason(self) -> str: + return self._reason + + def status(self) -> BackendStatus: + return BackendStatus(supported=False, platform_id=self._platform_id, reason=self._reason) + + def probe_permissions(self) -> PermissionProbe: + return PermissionProbe( + accessibility=PERMISSION_UNSUPPORTED, + screen_recording=PERMISSION_UNSUPPORTED, + responsible_hint="", + ) + + def _refuse(self) -> DriverResult: + """The one refusal every method returns.""" + return DriverResult( + ok=False, + text=REFUSAL_UNSUPPORTED.format(platform=self._platform_id, reason=self._reason), + ) + + def list_apps(self) -> DriverResult: + return self._refuse() + + def resolve_app(self, query: str) -> DriverResult: + return self._refuse() + + def snapshot(self, app: AppRef, req: SnapshotRequest) -> DriverResult: + return self._refuse() + + def click( + self, + app: AppRef, + rec: "ElementRec | None", + req: ClickRequest, + ) -> DriverResult: + return self._refuse() + + def drag(self, app: AppRef, req: DragRequest) -> DriverResult: + return self._refuse() + + def type_text(self, app: AppRef, rec: "ElementRec | None", text: str) -> DriverResult: + return self._refuse() + + def press_key(self, app: AppRef, rec: "ElementRec | None", key: str) -> DriverResult: + return self._refuse() + + def set_value(self, app: AppRef, rec: ElementRec, value: str) -> DriverResult: + return self._refuse() + + def scroll(self, app: AppRef, rec: ElementRec, direction: str, pages: float) -> DriverResult: + return self._refuse() + + def perform_action(self, app: AppRef, rec: ElementRec, action: str) -> DriverResult: + return self._refuse() + + def close(self) -> None: + """Nothing to release.""" + + +def unsupported_snapshot(app: AppRef) -> Snapshot: + """An empty snapshot for *app* — the shape an unsupported platform reports. + + Kept beside the refusal so a caller that needs a ``Snapshot`` object (rather + than a ``DriverResult``) never has to hand-build one and accidentally leave + ``has_secure`` unset. + """ + return Snapshot(app=app, elements=(), captured_at=0.0) + + +# ── Registry: one process-wide backend, swappable at runtime ── + +_shared_backend: ComputerUseBackend | None = None +_shared_backend_lock = threading.Lock() +_backend_factory: "Callable[[], ComputerUseBackend] | None" = None + + +def register_computer_use_backend(factory: "Callable[[], ComputerUseBackend] | None") -> None: + """Override the backend (the swap seam for tests and future platforms). + + Pass a factory returning a :class:`ComputerUseBackend`; the next + :func:`get_shared_backend` constructs through it. Pass ``None`` to restore + the platform default. Call :func:`reset_shared_backend` afterwards so an + already-built singleton — and the snapshot cache it filled — is replaced. + + The suite registers ``FakeComputerUseBackend`` process-wide so CI never + loads a native framework, never captures a real window, and never touches a + real application. + """ + global _backend_factory + with _shared_backend_lock: + _backend_factory = factory + + +def get_shared_backend() -> ComputerUseBackend: + """Process-wide backend singleton. + + One instance per process: a driver holds cached framework handles and an + event source, and building a second would double both for no benefit. + """ + global _shared_backend + with _shared_backend_lock: + if _shared_backend is None: + _shared_backend = (_backend_factory or select_default_backend)() + return _shared_backend + + +def reset_shared_backend() -> None: + """Drop the singleton AND the snapshot cache. + + Dropping the cache is not housekeeping — it is required for correctness. + Element indices are only meaningful against the walk that produced them, so + a new backend inheriting the previous one's snapshots could resolve an index + to a completely different element. + """ + global _shared_backend + with _shared_backend_lock: + if _shared_backend is not None: + try: + _shared_backend.close() + except Exception: + # A driver that fails to release its handles must not prevent the + # swap: leaving the old instance installed would be worse than + # leaking whatever it held. + logger.debug("computer-use backend close() failed", exc_info=True) + _shared_backend = None + index.reset_shared_index() + + +def select_default_backend() -> ComputerUseBackend: + """Build the backend for THIS platform. The only platform branch in the package. + + Branches on ``platform_compat.IS_MACOS`` / ``IS_WINDOWS`` / ``IS_LINUX`` + (never a raw ``sys.platform`` read), which is also what lets a test flip a + single flag and exercise the Windows/Linux degradation path on a Linux + runner. + + The native driver import is deferred into this function for two reasons: it + would be a circular import at module scope (the drivers subclass the classes + defined above), and a module-scope import would load ApplicationServices on + every machine that merely imports the package — including the Linux CI fleet, + where it would break collection of every test that transitively touches + ``kiro_crew``. + + A driver module that fails to import degrades to a typed refusal rather than + propagating: a partial install or a framework that will not load should + disable one capability, not crash the process that asked about it. + """ + if platform_compat.IS_MACOS: + try: + # Deferred + circular: macos_driver subclasses ComputerUseBackend. + from kiro_crew.computer_use.macos_driver import MacOSBackend + + return MacOSBackend() + except Exception as exc: + logger.warning("macOS computer-use driver unavailable: %s", exc) + return UnsupportedBackend(PLATFORM_MACOS, DRIVER_IMPORT_REASON.format(detail=exc)) + if platform_compat.IS_WINDOWS: + # Deferred + circular: windows_driver subclasses UnsupportedBackend. + from kiro_crew.computer_use.windows_driver import WindowsBackend + + return WindowsBackend() + if platform_compat.IS_LINUX: + # Deferred + circular: linux_driver subclasses UnsupportedBackend. + from kiro_crew.computer_use.linux_driver import LinuxBackend + + return LinuxBackend() + return UnsupportedBackend(PLATFORM_UNSUPPORTED, UNKNOWN_PLATFORM_REASON) + + +def platform_id_for_current_os() -> str: + """The platform id this OS would select, WITHOUT building a backend. + + For the dashboard payload and diagnostics: it answers "which platform is + this" without loading a native framework, which is why the Settings row can + render on any OS without a driver import. + """ + if platform_compat.IS_MACOS: + return PLATFORM_MACOS + if platform_compat.IS_WINDOWS: + return PLATFORM_WINDOWS + if platform_compat.IS_LINUX: + return PLATFORM_LINUX + return PLATFORM_UNSUPPORTED diff --git a/src/kiro_crew/computer_use/capture_macos.py b/src/kiro_crew/computer_use/capture_macos.py new file mode 100644 index 00000000000..7e39cb20d21 --- /dev/null +++ b/src/kiro_crew/computer_use/capture_macos.py @@ -0,0 +1,294 @@ +"""Window capture: pixels to a persisted, size-bounded JPEG. Fully in-process. + +**No subprocess and no image library.** Not ``/usr/sbin/screencapture``, not +Pillow. ``CGWindowListCreateImage`` grabs one window's pixels and ImageIO's +``CGImageDestination`` encodes them, with ImageIO performing the downscale itself +via ``kCGImageDestinationImageMaxPixelSize``. Three liabilities disappear with +that choice: there is no spawn node for the spawn audit to account for, no +optional dependency to degrade around (Pillow is declared in neither +``setup.cfg`` nor ``pyproject.toml``), and no 204ms process launch per capture. + +Why the image is a PATH and never bytes. The MCP transport +(``validation.build_tool_response``) emits ``{"content":[{"type":"text",...}]}`` +and cannot express an image block, so a relayed screenshot would have to be +base64 in text — measured at ~41,000 tokens for a single raw window PNG. The +compressed file goes to disk and only its path is relayed, which the model reads +with ``fs_read`` if and only if the accessibility tree was insufficient. At +1280px/q0.55 a real window encodes to ~24KB (~8,300 tokens if read at all), and +the image is corroboration rather than the primary channel. + +Size reporting comes from :func:`macos_ffi.jpeg_dimensions` — the ENCODED +dimensions, parsed back out of the JPEG. ``CGImageGetWidth`` on the *input* image +would over-report by the downscale factor (verified: a 1676x1320 window encodes to +1280x1008), and a wrong size in the result would make a model reason about a +resolution it is not going to get. + +The encoded frame has exactly one other consumer: :mod:`screencast` relays these +same bytes to the dashboard's floating live view. It is a relay, not a second +capture — no extra ``CGWindowListCreateImage`` call, no timer, no full-screen +grab — and it re-checks ``has_secure`` plus the permitted screenshot channel +before anything leaves the process. + +Accepted residual risk, stated here because a reviewer will find it: the persisted +JPEGs live in a ``0o700`` temp dir the agent can reach with ``fs_read``. That is +the posture the browse module already ships. Computer use widens WHAT can be in +frame (any window, not one browser tab), which is why capture is per-window and +never full-screen, why a window holding any secure field is not captured at all, +and why the directory is ring-trimmed. It is not claimed to be closed. +""" + +from __future__ import annotations + +import logging +import os +import tempfile +import time +from dataclasses import replace + +from kiro_crew import platform_compat +from kiro_crew.computer_use import macos_ffi, screencast +from kiro_crew.computer_use.types import ( + DEFAULT_SCREENSHOT_JPEG_QUALITY, + DEFAULT_SCREENSHOT_MAX_PX, + MAX_SCREENSHOT_MAX_PX, + MIN_SCREENSHOT_MAX_PX, + SCREENSHOT_FILE_PREFIX, + SCREENSHOT_FILE_SUFFIX, + SCREENSHOT_KEEP, + AppRef, + Snapshot, +) + +logger = logging.getLogger(__name__) + +# Owner-only. The directory holds pixels of the operator's own windows, so no +# other local account may list or read it. +_SHOT_DIR_MODE = 0o700 +# ImageIO takes quality as 0.0-1.0; the config field is the 0-100 integer users +# and the dashboard understand. +_QUALITY_SCALE = 100.0 +# Millisecond timestamp in the filename: two captures inside the same second are +# routine (a get_state followed by a mutating action's re-snapshot) and a +# second-resolution name would collide and silently overwrite. +_TIMESTAMP_SCALE = 1000 + + +def shots_dir() -> str: + """Path to the screenshot directory (no side effects). + + Resolved through :func:`macos_ffi.shots_dir_default`, which builds it from + ``tempfile.gettempdir()`` rather than a hardcoded ``/tmp`` — the same idiom + ``mcp_playwright_proxy`` uses, and the reason a Windows port would not need an + edit here. + """ + return macos_ffi.shots_dir_default() + + +def ensure_shots_dir() -> str: + """Create the screenshot directory ``0o700`` and return its path. + + ``mode=`` on ``makedirs`` is applied by the OS through the umask, so the mode + is re-asserted with :func:`platform_compat.chmod_safe` afterwards. Without + that a permissive umask would leave the directory group- or world-readable and + the whole point of the mode would be lost. + """ + path = shots_dir() + os.makedirs(path, mode=_SHOT_DIR_MODE, exist_ok=True) + try: + platform_compat.chmod_safe(path, _SHOT_DIR_MODE) + except OSError: + # Warn and continue: a directory whose mode cannot be tightened is a real + # concern the operator should see, but refusing to capture would disable a + # feature over a filesystem that may not support modes at all. + logger.warning("could not restrict computer-use screenshot dir %s to owner-only", path) + return path + + +def capture_snapshot_image( + snap: Snapshot, + *, + max_px: int = DEFAULT_SCREENSHOT_MAX_PX, + quality: int = DEFAULT_SCREENSHOT_JPEG_QUALITY, +) -> Snapshot: + """Capture *snap*'s window, persist the JPEG, and return an updated snapshot. + + Returns the snapshot UNCHANGED (no image) when: + + * the snapshot contains any secure element — the always-on floor. A password + field's rendered glyphs are a credential even though the tree redacted its + value, and there is no reliable way to blank a sub-rectangle of an + already-encoded JPEG, so suppression is whole-window. The renderer says so + explicitly rather than silently omitting the line, because a model that + asked for pixels and got none retries in a loop unless it is told the + omission was deliberate; + * the window id is unknown; + * the capture or encode produced no bytes (a closed window yields a NULL image + — verified, not a crash); + * persisting failed. + + Never raises. The accessibility tree is the primary channel, so a capture + failure must degrade the result rather than fail the observation. + """ + if snap.has_secure: + return snap + # FAIL CLOSED on an incomplete scan. ``has_secure`` is set for every node the + # walk CLASSIFIED — which now includes nodes past the reporting budget — but the + # walk still has hard cutoffs of its own (``MAX_TREE_NODES_LIMIT`` nodes and the + # ``MAX_WALK_SECS`` deadline). If either fired, the walk never reached the end of + # the window, so ``has_secure=False`` means "none seen", NOT "none present": a + # password field beyond the cutoff would leave rendered credentials capturable. + # + # "Unknown" therefore has to behave like "present" here. This is the one gate + # that decides whether pixels leave the process, and the cost of being wrong in + # each direction is not symmetric — a suppressed screenshot costs the model one + # announced omission (the renderer says so, so it does not retry blindly), while + # a wrong capture photographs somebody's password box. + if snap.truncated or snap.depth_truncated: + return snap + if snap.app.window_id <= 0: + return snap + + raw, width, height = _encode_window(snap.app, max_px=max_px, quality=quality) + if not raw: + return snap + + path = persist_jpeg(raw) + if not path: + return snap + # ``dataclasses.replace``, NOT a field-by-field rebuild. Enumerating the fields + # here made this function silently lossy: every field added to ``Snapshot`` + # afterwards was dropped whenever a screenshot was attached, and the shape of the + # loss is what made it nasty — the SAME snapshot without a screenshot carried + # them fine, so it would only misbehave on responses that also carry an image. + # (Caught with ``window_bounds``/``selected_text``, whose absence would delete the + # element-frame origin line from exactly those responses.) ``replace`` cannot go + # stale; ``test_attaching_a_screenshot_preserves_EVERY_other_snapshot_field`` + # pins it field-by-field over the whole dataclass. + captured = replace( + snap, + image_jpeg=raw, + image_path=path, + image_width=width, + image_height=height, + ) + # Mirror the frame we JUST encoded to the dashboard's live view. Nothing is + # captured for the mirror — it relays these exact already-downscaled bytes — + # and ``emit_snapshot_frame`` owns all three suppressions (no published + # surface scope, a secure window, a withheld screenshot channel). It does not + # block: the POST runs on a daemon thread, so a dead + # gateway cannot slow the observation the model asked for. + # + # Wrapped anyway, even though the relay is itself contracted never to raise: + # this function's OWN contract is "never raises", and a decorative mirror must + # not be able to turn a successful observation into a failed tool call if that + # inner contract is ever broken. + try: + screencast.emit_snapshot_frame(captured) + except Exception: + logger.debug("computer-use live-view relay failed", exc_info=True) + return captured + + +def _encode_window(app: AppRef, *, max_px: int, quality: int) -> tuple[bytes, int, int]: + """Capture + encode one window. Returns ``(bytes, width, height)``. + + Clamps *max_px* and *quality* here rather than trusting the caller: the MCP + schemas validate agent input, but this function is also reachable from config, + and a zero or negative ``max_px`` handed to ImageIO would produce either a + degenerate image or an unbounded one. + """ + clamped_px = max(MIN_SCREENSHOT_MAX_PX, min(int(max_px), MAX_SCREENSHOT_MAX_PX)) + clamped_quality = max(1, min(int(quality), 100)) / _QUALITY_SCALE + try: + return macos_ffi.capture_window_jpeg( + app.window_id, max_px=clamped_px, quality=clamped_quality + ) + except Exception: + logger.debug("window capture failed for %s", app.label, exc_info=True) + return b"", 0, 0 + + +def persist_jpeg(raw: bytes) -> str: + """Write *raw* into the screenshot dir and return its path, or ``""``. + + Owner-only on the file as well as the directory + (:func:`platform_compat.restrict_to_owner`, which is fail-loud): defence in + depth for the case where the directory mode could not be applied. + + Ring-trims after writing so the directory is bounded whatever happens next — + trimming first would leave the cap violated by exactly one file for the + lifetime of a session that then crashed. + """ + if not raw: + return "" + try: + directory = ensure_shots_dir() + # ATOMIC unique allocation, not a millisecond timestamp. The gateway + # offloads snapshots to a thread pool, so two captures of DIFFERENT apps can + # land in the same millisecond; a timestamp-only name then resolves to one + # path and both writers open it — the second overwrites the first and one + # caller is handed a screenshot of an application it never asked about + # (a cross-app pixel leak, not merely a lost file). + # + # ``mkstemp`` also creates the file 0o600 from the outset, so there is no + # window in which it exists world-readable before ``restrict_to_owner`` + # runs. The timestamp stays in the prefix because the ring trim orders by + # mtime and a human reading the spool wants it. + prefix = f"{SCREENSHOT_FILE_PREFIX}{int(time.time() * _TIMESTAMP_SCALE)}-" + handle_fd, path = tempfile.mkstemp( + prefix=prefix, suffix=SCREENSHOT_FILE_SUFFIX, dir=directory + ) + with os.fdopen(handle_fd, "wb") as handle: + handle.write(raw) + except OSError: + logger.warning("could not persist computer-use screenshot", exc_info=True) + return "" + try: + platform_compat.restrict_to_owner(path) + except OSError: + # Warn and continue — the same posture every other secret-bearing writer + # in this repo takes. The file is already inside a 0o700 directory. + logger.warning("could not restrict computer-use screenshot %s to owner-only", path) + trim_shots_dir() + return path + + +def trim_shots_dir(keep: int = SCREENSHOT_KEEP) -> int: + """Delete all but the newest *keep* screenshots. Returns the number removed. + + The directory is a cache, not an archive: a long session must not be able to + fill the temp volume. Ordered by filename rather than by mtime — the names + carry a millisecond timestamp, so a lexical sort IS chronological and needs no + ``stat`` per file. + + Never raises: a file another process removed concurrently, or one we cannot + delete, is skipped. + """ + if keep <= 0: + return 0 + directory = shots_dir() + try: + names = sorted( + name + for name in os.listdir(directory) + if name.startswith(SCREENSHOT_FILE_PREFIX) and name.endswith(SCREENSHOT_FILE_SUFFIX) + ) + except OSError: + return 0 + removed = 0 + for name in names[: max(0, len(names) - keep)]: + try: + os.unlink(os.path.join(directory, name)) + removed += 1 + except OSError: + logger.debug("could not trim screenshot %s", name, exc_info=True) + return removed + + +__all__ = [ + "capture_snapshot_image", + "ensure_shots_dir", + "persist_jpeg", + "shots_dir", + "trim_shots_dir", +] diff --git a/src/kiro_crew/computer_use/cli.py b/src/kiro_crew/computer_use/cli.py new file mode 100644 index 00000000000..22d102928cc --- /dev/null +++ b/src/kiro_crew/computer_use/cli.py @@ -0,0 +1,485 @@ +"""``kirocrew computer`` — human-facing computer-use diagnostics and a debug driver. + +Three subcommands: + +* ``doctor [--json]`` — platform support, the keystone primary enable, and the + ADVISORY macOS permission probe. ``--json`` is the machine form the dashboard + shells out for its Settings permission rows, which is how the native + accessibility probe stays OUT of the gateway process: a ctypes fault in a + short-lived subprocess costs one diagnostic, while the same fault in-gateway + would take down every session, the cron scheduler, the Slack socket and every + dashboard websocket at once. +* ``apps`` — the on-screen application list, so a user can see the names and + bundle ids the agent will match against. +* ``call`` — run ONE computer-use tool, or a JSON array of them **in a single + process**, against the same dispatch chokepoint the agent uses. See below. + +Deliberately NO ``kirocrew computer state ``. That would be a second, +CLI-shaped spelling of an LLM-facing capability, and the MCP-first rule requires +an LLM-facing capability to be an MCP tool (it is: ``computer_get_state``). +``doctor`` is a permission diagnostic rather than a capability, and ``apps`` has +its MCP twin (``computer_list_apps``), so neither brushes the rule. + +**Why ``call`` does not brush the MCP-first rule either.** It is not a new +capability and it adds no tool: it is a *harness* over the existing ten, for a +human reproducing a failure at a terminal. The rule exists so the model gets a +structured tool rather than being told to shell out — and the model already has +all ten as MCP tools. ``call`` deliberately has no MCP twin, because a tool that +runs other tools would let a model launder a per-call gate decision through one +approved invocation. + +**``call`` is fully gated, and that is the point of routing it through +``tools.dispatch_tool`` rather than reaching into ``service``.** Every call goes +through the same ordered chokepoint as an agent call: the keystone primary enable, +the fail-closed ``gate.require_computer_use``, the built-in app denylist, index +freshness, the secure-target refusals, and the observation ceiling. So this +command cannot be used to see or do anything the agent could not, which is +exactly what makes it a faithful reproduction tool. Two consequences worth +stating rather than discovering: + +* the session key is the attended CLI surface (:data:`_CLI_SESSION_KEY`) — a real + surface the gate accepts, not a bypass sentinel; +* ``approval_recorded`` is left at ``False``. The approval ceiling it used to + satisfy is gone, so the flag no longer changes any outcome — but it is still + never minted here, because doing so would be the CLI asserting a prompt that + nobody answered on this leg (asserted by an AST test over the whole package). + +**Why a whole array in one process.** ``element_index`` values only mean anything +relative to the ``computer_get_state`` that produced them, and the cache that +holds that mapping (``index.SnapshotIndex``, reached via the shared service +singleton) is per-process with a 90s TTL. Two ``kirocrew computer call`` +invocations therefore cannot share indices at all — the second would refuse with +"no state for …". ``--calls`` exists so a snapshot-then-act sequence is +reproducible from one command line, which is the shape the reference +implementation's ``call --calls`` has for the same reason. + +Hand-rolled dispatch mirroring ``browser/cli.py`` rather than argparse +subparsers: the command surface is two words plus free-form arguments, and the +parent CLI already forwards ``REMAINDER``. +""" + +from __future__ import annotations + +import json +import logging +import sys +from typing import Any, Mapping + +from kiro_crew.computer_use import enable_state, service, tools +from kiro_crew.computer_use.backend import platform_id_for_current_os +from kiro_crew.computer_use.policy import blocked_app_categories +from kiro_crew.computer_use.types import ( + ERROR_PREFIX, + PERMISSION_UNKNOWN, + STATE_KEY_ENABLED, + TOOL_LIST_APPS, +) + +logger = logging.getLogger(__name__) + +# Exit codes. ``doctor`` reports a real status, so a script can gate on it; the +# JSON form always exits 0 because its consumer reads the body. +_EXIT_OK = 0 +_EXIT_PROBLEM = 1 + +# The surface identity ``call`` presents to the gate. ``cli_chat`` is the repo's +# existing key for "a human at a terminal" — ``sel._infer_source`` maps it to the +# ``cli`` surface, which is deliberately NOT in ``gate.UNATTENDED_SURFACES`` and +# matches none of ``gate.UNATTENDED_KEY_PREFIXES``, so the unattended-surface +# refusal does not fire. Reusing it (rather than minting a private key) is what +# makes an operator profile bound to the CLI surface govern this command too. +_CLI_SESSION_KEY = "cli_chat" + + +def _session_key() -> str: + """The identity to gate this invocation with — always the attended CLI surface. + + ``cli_chat`` is the repo's existing key for "a human at a terminal" + (``sel._infer_source`` maps it to the ``cli`` surface). It is used + unconditionally now: the unattended-surface refusal that used to make this + decision load-bearing is gone, so there is nothing left for a stricter identity + to buy. The key still matters for the SEL audit trail, which is why this is a + named surface rather than an empty string. + """ + return _CLI_SESSION_KEY + + +# Shown when an unauthenticated invocation has no keystone opt-in. Names the flag +# and the file, because "refused" without the remedy is not a usable diagnostic. + +# ``--calls`` entry keys. A batch entry is ``{"tool": "...", "args": {...}}``; +# ``args`` is optional so a no-argument tool is ``{"tool": "computer_end_turn"}``. +_CALL_KEY_TOOL = "tool" +_CALL_KEY_ARGS = "args" +_CALL_KEYS: frozenset[str] = frozenset({_CALL_KEY_TOOL, _CALL_KEY_ARGS}) + +_USAGE = """kirocrew computer — desktop automation (computer use) diagnostics + +Commands: + doctor Show platform support, whether computer use is enabled, and + the macOS Accessibility / Screen Recording permission hints + doctor --json The same report as JSON (used by the dashboard) + apps List applications with an on-screen window + call [key=value ...] + Run one computer-use tool and print its reply + call --calls '[{"tool": "...", "args": {...}}, ...]' + Run several tools in ONE process, so element_index values from + an earlier computer_get_state are still valid for later calls + call ... --json Emit the replies as a JSON array instead of prose + +A key=value argument is parsed as JSON when it can be (element_index=3, +screenshot=false, x=120.5) and kept as a plain string otherwise (app=Finder). +Wrap a value with spaces in shell quotes: text='hello there'. + +Every call runs through the same gate the agent does — the primary enable, +security policy, the app denylist and the secure-field refusals all apply, so +this cannot reach anything the agent could not. Under a policy that forces +interactive approval, a mutating call is refused here: there is no prompt on this +leg for anyone to answer. + +Computer use is OFF by default and can only be enabled by you, from the +dashboard: Settings -> Computer Use. An agent cannot enable it. +""" + + +def main() -> None: + """Console entry point (``kirocrew computer ...``).""" + run_computer(sys.argv[1:]) + + +def run_computer(args: list[str]) -> None: + """Entry point for ``kirocrew computer ``.""" + if not args: + print(_USAGE) + return + cmd = args[0] + if cmd in ("-h", "--help", "help"): + print(_USAGE) + return + if cmd == "doctor": + _cmd_doctor(as_json="--json" in args[1:]) + return + if cmd == "apps": + _cmd_apps() + return + if cmd == "call": + _cmd_call(args[1:]) + return + print(f"Unknown command: {cmd}. Run 'kirocrew computer' for help.", file=sys.stderr) + sys.exit(_EXIT_PROBLEM) + + +def _cmd_doctor(*, as_json: bool) -> None: + """Print the support/enable/permission report.""" + report = build_doctor_report() + if as_json: + # Exit 0 regardless: the dashboard reads the body, and a non-zero exit + # would make it treat a legitimately-unsupported platform as a probe + # failure and render "unknown" instead of the real reason. + print(json.dumps(report, indent=2)) + return + + print(f"Platform: {report['platform']}") + if report["supported"]: + print("Supported: yes") + else: + print(f"Supported: no — {report['reason']}") + print(f"Enabled: {'yes' if report['enabled'] else 'no (Settings -> Computer Use)'}") + + perms = report["permissions"] + if report["platform"] == "macos": + print(f"Accessibility: {perms['accessibility']}") + print(f"Screen record: {perms['screen_recording']}") + if perms.get("responsible_hint"): + print(f"Grant to: {perms['responsible_hint']}") + # Stated every time, not only on a miss. macOS attributes a TCC grant to + # the RESPONSIBLE PARENT of the process tree, so a probe reporting + # "missing" while a full-fidelity capture succeeds is normal — and a user + # who believes the probe will chase a permission they already have. + print() + print( + "Note: these permission readings are advisory. macOS attributes a\n" + "grant to the process that launched KiroCrew, so 'missing' does not\n" + "always mean unavailable — and computer use is never gated on them." + ) + + if report["blocked_apps"]: + print() + print("Always-blocked targets (built in, not configurable):") + for entry in report["blocked_apps"]: + print(f" - {entry['category']}: {entry['reason']}") + + if report["errors"]: + print() + for message in report["errors"]: + print(f"Problem: {message}", file=sys.stderr) + sys.exit(_EXIT_PROBLEM) + sys.exit(_EXIT_OK if report["supported"] else _EXIT_PROBLEM) + + +def build_doctor_report() -> dict: + """Assemble the doctor report. Never raises. + + Every probe is individually guarded and its failure recorded in ``errors``: + this function is what the dashboard shells out to, and it must produce a + usable payload on a machine where the accessibility framework will not load + at all. A missing probe degrades to ``unknown``, never to ``granted``. + """ + errors: list[str] = [] + platform_id = platform_id_for_current_os() + + enabled = False + try: + enabled = enable_state.is_enabled() + except Exception as exc: + errors.append(f"could not read the computer-use state file: {exc}") + + supported = False + reason = "" + permissions = { + "accessibility": PERMISSION_UNKNOWN, + "screen_recording": PERMISSION_UNKNOWN, + "responsible_hint": "", + } + try: + svc = service.get_shared_service() + status = svc.status() + supported = status.supported + reason = status.reason + platform_id = status.platform_id or platform_id + probe = svc.probe_permissions() + permissions = { + "accessibility": probe.accessibility, + "screen_recording": probe.screen_recording, + "responsible_hint": probe.responsible_hint, + } + except Exception as exc: + # A driver that will not even construct is exactly what this command + # exists to report, so name it rather than crashing. + errors.append(f"the computer-use driver could not be probed: {exc}") + reason = reason or str(exc) + + return { + "platform": platform_id, + "supported": supported, + "reason": reason, + "enabled": enabled, + "state_file": str(_state_path()), + "state_key": STATE_KEY_ENABLED, + "permissions": permissions, + "blocked_apps": [dict(entry) for entry in blocked_app_categories()], + "errors": errors, + } + + +def _state_path() -> object: + """The keystone state path, or a placeholder when it cannot be resolved.""" + try: + return enable_state.computer_use_state_path() + except Exception: + return "(unresolved)" + + +def _cmd_apps() -> None: + """Print the on-screen application list, through the SAME gate as ``call``. + + Reviewer finding: this used to call ``service.list_apps()`` directly, on the + reasoning that a diagnostic in the operator's own terminal is not the agent. + That reasoning does not hold, because the agent can run this command with + bash — so the direct call was an ungated read of every window TITLE (document + names, paths, and whatever a terminal put in its title) that worked even with + the feature disabled, in an unattended cron session, or under a policy that + bans computer use outright. + + Routing it through ``dispatch_tool`` costs the operator nothing they should + have had: ``computer_list_apps`` renders the same list, filtered by the app + denylist and the observation ceiling. If the answer is a refusal, that refusal + IS the diagnostic — and ``kirocrew computer doctor`` (which reads only the + keystone and the TCC state, never the window list) is still the ungated way to + find out why the feature is off. + """ + print(tools.dispatch_tool(TOOL_LIST_APPS, {}, session_key=_session_key())) + + +def _cmd_call(argv: list[str]) -> None: + """Run one tool, or a ``--calls`` batch, and print the replies. + + Exits non-zero when ANY reply is an error, so a shell can gate on it. The + check is the ``Error: `` prefix rather than an exception, because + ``dispatch_tool`` is contracted to return every refusal as text — the prefix + is the same load-bearing marker ``mcp_shared.call_tool_with_logging`` + classifies a failed SEL outcome from. + + Batch execution is SEQUENTIAL and does **not** stop at the first error. A + reproduction is more useful whole: seeing that step 2 was refused and step 3 + then hit a stale index is the actual story, whereas aborting would hide the + second half of it. + """ + as_json = "--json" in argv + rest = [arg for arg in argv if arg != "--json"] + + try: + calls = _parse_calls(rest) + except ValueError as exc: + print(f"Error: {exc}", file=sys.stderr) + print(file=sys.stderr) + print(_USAGE, file=sys.stderr) + sys.exit(_EXIT_PROBLEM) + + # AFTER parsing (so a malformed command still teaches the syntax) but before any + # dispatch: an unauthenticated caller with no keystone opt-in gets ONE clear + # refusal rather than the same one repeated for every entry of a batch. + + # Resolved ONCE for the whole batch: re-reading the keystone per call would let a + # concurrent Settings change split one batch across two ceilings. + session_key = _session_key() + results: list[dict[str, Any]] = [] + for tool_name, tool_args in calls: + # ``dispatch_tool`` never raises except for ``PlatformCompositionError`` + # (a host that cannot compose its platform context must NOT degrade to a + # text refusal), so that one propagates out of this command too and the + # rest of the batch is abandoned — an un-composable ceiling is not a + # per-call failure. + text = tools.dispatch_tool(tool_name, tool_args, session_key=session_key) + results.append({_CALL_KEY_TOOL: tool_name, "text": text}) + + failed = any(str(entry["text"]).startswith(ERROR_PREFIX) for entry in results) + if as_json: + print(json.dumps(results, indent=2)) + else: + for position, entry in enumerate(results): + if len(results) > 1: + # Only label in a batch: a single call's reply should be pasteable + # as-is, and a header would corrupt a copy of the rendered tree. + print(f"── {position + 1}/{len(results)} {entry[_CALL_KEY_TOOL]} ──") + print(entry["text"]) + if failed: + sys.exit(_EXIT_PROBLEM) + + +def _parse_calls(argv: list[str]) -> list[tuple[str, dict[str, Any]]]: + """Turn the argv tail into an ordered ``(tool_name, args)`` list. + + Two accepted forms, and never a mix — ``--calls`` carries its own tool names, + so a positional tool beside it is ambiguous about ordering and is rejected + rather than guessed at: + + * ``--calls ''`` — the batch form; + * `` [key=value ...]`` — the single form. + + Raises :class:`ValueError` with a user-facing sentence for every malformed + input. Argument TYPES are deliberately not checked here: the per-tool schema + in ``MCP_COMPUTER_SCHEMAS`` is the one authority on them, and re-stating it + would create a second, drifting copy. This function only produces the shape + ``dispatch_tool`` takes. + """ + if not argv: + raise ValueError("call needs a tool name, or --calls with a JSON array") + + if argv[0] == "--calls": + if len(argv) < 2: + raise ValueError("--calls needs a JSON array argument") + if len(argv) > 2: + raise ValueError("--calls takes exactly one JSON array; quote it as one argument") + return _parse_batch(argv[1]) + if "--calls" in argv: + raise ValueError("use either --calls or a positional tool name, not both") + + tool_name = argv[0] + if tool_name.startswith("-"): + raise ValueError(f"expected a tool name, got the flag {tool_name!r}") + return [(tool_name, _parse_kv_args(argv[1:]))] + + +def _parse_batch(raw: str) -> list[tuple[str, dict[str, Any]]]: + """Parse the ``--calls`` JSON array into ``(tool_name, args)`` pairs.""" + try: + parsed = json.loads(raw) + except json.JSONDecodeError as exc: + raise ValueError(f"--calls is not valid JSON: {exc}") from exc + if not isinstance(parsed, list): + raise ValueError('--calls must be a JSON array of {"tool": ..., "args": ...} objects') + if not parsed: + raise ValueError("--calls array is empty") + + calls: list[tuple[str, dict[str, Any]]] = [] + for position, entry in enumerate(parsed): + label = f"--calls entry {position + 1}" + if not isinstance(entry, dict): + raise ValueError(f"{label} must be a JSON object") + unknown = sorted(set(entry) - _CALL_KEYS) + if unknown: + # Rejected rather than ignored: a misspelled ``arg``/``arguments`` key + # would otherwise run the tool with NO arguments, which for a real + # desktop action is a silently different call, not a no-op. + raise ValueError(f"{label} has unknown key(s): {', '.join(unknown)}") + tool_name = entry.get(_CALL_KEY_TOOL) + if not isinstance(tool_name, str) or not tool_name: + raise ValueError(f"{label} needs a non-empty string 'tool'") + tool_args = entry.get(_CALL_KEY_ARGS, {}) + if tool_args is None: + tool_args = {} + if not isinstance(tool_args, dict): + raise ValueError(f"{label} 'args' must be a JSON object") + calls.append((tool_name, dict(tool_args))) + return calls + + +def _parse_kv_args(tokens: list[str]) -> dict[str, Any]: + """Parse ``key=value`` tokens, JSON-decoding each value when possible. + + ``element_index=3`` must arrive as an int and ``screenshot=false`` as a bool, + because the schema rejects the string forms — so a bare ``json.loads`` is + tried first. It is a FALLBACK, not a requirement: ``app=Finder`` is not valid + JSON and must stay the string ``"Finder"``, and ``text=null`` staying the + four-character string it looks like at a shell prompt is the less surprising + reading of a typed argument. Quote a value to force the string form + (``app='"Finder"'`` is the same thing either way). + """ + parsed: dict[str, Any] = {} + for token in tokens: + if token.startswith("-"): + raise ValueError(f"unknown flag {token!r}; arguments are key=value") + key, sep, raw = token.partition("=") + if not sep or not key: + raise ValueError(f"expected key=value, got {token!r}") + if key in parsed: + raise ValueError(f"{key} given twice") + parsed[key] = _coerce(raw) + return parsed + + +def _coerce(raw: str) -> Any: + """JSON-decode *raw* when it parses to a non-string scalar, else keep the text. + + A JSON string/array/object literal is kept as TEXT on purpose: a value that + already decodes to a ``str`` gains nothing from the round trip, and letting a + ``key=[1,2]`` build a real list would hand the schema a container no + computer-use field accepts, turning a typo into a confusing type error rather + than a plain "unknown field"/length refusal. + """ + try: + value = json.loads(raw) + except (json.JSONDecodeError, ValueError): + return raw + if isinstance(value, (bool, int, float)) and not isinstance(value, str): + return value + return raw + + +def run_calls(calls: "list[tuple[str, Mapping[str, Any]]]") -> list[str]: + """Run *calls* in this process and return their replies, in order. + + The programmatic form of ``call --calls``, exposed for a test (and for a future + harness) so the batch semantics — one process, one snapshot cache, sequential, + no abort on error — can be exercised without capturing stdout or catching + :func:`sys.exit`. + + ``PlatformCompositionError`` propagates, exactly as it does through + ``dispatch_tool``. + """ + session_key = _session_key() + return [tools.dispatch_tool(name, dict(args), session_key=session_key) for name, args in calls] + + +__all__ = ["build_doctor_report", "main", "run_calls", "run_computer"] diff --git a/src/kiro_crew/computer_use/cursor_motion.py b/src/kiro_crew/computer_use/cursor_motion.py new file mode 100644 index 00000000000..aff8402e655 --- /dev/null +++ b/src/kiro_crew/computer_use/cursor_motion.py @@ -0,0 +1,425 @@ +"""The Cursor Motion PATH MODEL — pure geometry, zero platform contact. + +This module is deliberately the boring half of Cursor Motion: given a start and +an end point it produces a **sampled list of screen points** plus a duration, and +that is all. No ctypes, no subprocess, no AppKit, no config read, no clock beyond +what the caller passes in. Everything here is a total function of its arguments, +which is what makes the visual behaviour of the feature unit-testable on a Linux +CI shard with no display at all. + +The split is load-bearing. ``overlay_proc`` (AppKit, out of process) and +``overlay`` (the gateway-side supervisor) are the parts that can fail for +environmental reasons; keeping the *shape* of the motion here means a regression +in how the cursor moves is caught by an assertion on numbers rather than by +somebody watching a screen. + +The model, reimplemented from the reference project's ``CursorMotionModel.swift`` +(read for the algorithm, not copied): + +* **One cubic Bezier** from ``start`` to ``end``, bowed sideways by a + perpendicular *arc* whose magnitude is ``clamp(distance * 0.22, 28, 110) * + curve_scale``. A straight interpolation reads as a teleport and is exactly what + makes a synthetic cursor look synthetic; the arc is the whole illusion. +* **Asymmetric control points** (0.18/0.10 and 0.80/0.96 along/across the chord) + so the cursor leaves quickly and arrives settling. ``curve_scale == 0`` + collapses the handles to the classic thirds placement, i.e. an exact straight + line — the escape hatch for "I want no flourish". +* **A progress spring** (velocity-Verlet, fixed 1/240s step, + ``response=1.4``/``damping=0.9``) integrated from 0 to 1 and then fed through + the Bezier. Because the spring drives PROGRESS rather than position, its slight + numerical overshoot past 1.0 is *clamped* before sampling, so the drawn cursor + can never overshoot its target position — an important property when the point + of the animation is to show the user where a click is about to land. + +Coordinate convention: every point in this module is in **top-left screen +coordinates** (y grows downward), because that is what the rest of computer use +uses — the AX/CG surfaces, ``screencapture -R`` and the element frames all agree +on it. The bottom-left flip that AppKit's ``NSWindow`` origin needs happens in +``overlay_proc``, at the one place that actually talks to AppKit, and nowhere +else. +""" + +from __future__ import annotations + +import logging +import math +from dataclasses import dataclass + +from kiro_crew.computer_use.types import ( + CURVE_AMOUNT_MAX, + CURVE_AMOUNT_MIN, + CURVE_C1_ACROSS, + CURVE_C1_ALONG, + CURVE_C2_ACROSS, + CURVE_C2_ALONG, + CURVE_C2_OFFSET_RATIO, + CURVE_DISTANCE_RATIO, + DEFAULT_CURVE_SCALE, + DEFAULT_PATH_SAMPLES, + FULL_SPEED_DISTANCE, + MAX_CURVE_SCALE, + MAX_MOVE_DURATION_MS, + MAX_PATH_SAMPLES, + MIN_MOTION_DISTANCE, + MIN_MOVE_DURATION_MS, + MIN_PATH_SAMPLES, + MOTION_EPSILON, + SPRING_DAMPING_FRACTION, + SPRING_DT, + SPRING_FALLBACK_SETTLE_SECS, + SPRING_MAX_STEPS, + SPRING_MAX_STIFFNESS, + SPRING_RESPONSE, + SPRING_SETTLE_DISTANCE, + STRAIGHT_C1_FRACTION, + STRAIGHT_C2_FRACTION, + STRAIGHT_MOVE_DISTANCE, +) + +logger = logging.getLogger(__name__) + +__all__ = [ + "CursorPath", + "MotionPlan", + "SpringConfig", + "build_path", + "plan_motion", + "sample_path", + "settle_time", + "spring_progress_curve", +] + + +@dataclass(frozen=True) +class SpringConfig: + """The progress spring's two tunables plus everything derived from them. + + ``stiffness`` and ``drag`` are computed in :meth:`create` rather than stored + by hand so the pair can never disagree with ``response``/``damping``: a + hand-set drag that does not match the stiffness is precisely how a "settle" + animation turns into a visible bounce or an eternal crawl. + """ + + response: float = SPRING_RESPONSE + damping: float = SPRING_DAMPING_FRACTION + dt: float = SPRING_DT + stiffness: float = 0.0 + drag: float = 0.0 + + @classmethod + def create( + cls, + *, + response: float = SPRING_RESPONSE, + damping: float = SPRING_DAMPING_FRACTION, + dt: float = SPRING_DT, + ) -> "SpringConfig": + """Build a config with ``stiffness``/``drag`` derived from *response*. + + Every input is floored: a non-positive ``response`` would make stiffness + infinite (``(2*pi/0)**2``) and the first integrator step NaN, and a + non-positive ``dt`` would loop forever. Both are clamped rather than + rejected, because this is a cosmetic subsystem and a caller passing a + silly number should get an ugly animation, never an exception on the path + of a tool call. + """ + safe_response = max(float(response), MOTION_EPSILON) + safe_dt = max(float(dt), MOTION_EPSILON) + stiffness = min((2.0 * math.pi / safe_response) ** 2, SPRING_MAX_STIFFNESS) + drag = 2.0 * max(float(damping), 0.0) * math.sqrt(stiffness) + return cls( + response=safe_response, + damping=max(float(damping), 0.0), + dt=safe_dt, + stiffness=stiffness, + drag=drag, + ) + + +@dataclass(frozen=True) +class CursorPath: + """A cubic Bezier in top-left screen coordinates. + + Holding the control points (rather than only the sampled result) keeps the + curve re-samplable at a different density and makes the geometry directly + assertable in a test — the arc offset is the one number whose regression a + human would notice and no other assertion would catch. + """ + + start: tuple[float, float] + end: tuple[float, float] + control1: tuple[float, float] + control2: tuple[float, float] + curve_scale: float = DEFAULT_CURVE_SCALE + arc_amount: float = 0.0 + + def point_at(self, t: float) -> tuple[float, float]: + """The curve point at parameter *t*, clamped to ``[0, 1]``. + + Clamped rather than extrapolated: a ``t`` past 1 on a cubic Bezier flies + off along the end tangent, which for a fake cursor means jumping past the + thing it is supposed to be pointing at. The endpoints are returned + EXACTLY (not evaluated) so ``point_at(0)``/``point_at(1)`` are bit-equal + to ``start``/``end`` and cannot drift by a float epsilon. + """ + clamped = 0.0 if t < 0.0 else (1.0 if t > 1.0 else float(t)) + if clamped <= 0.0: + return self.start + if clamped >= 1.0: + return self.end + omt = 1.0 - clamped + a = omt * omt * omt + b = 3.0 * omt * omt * clamped + c = 3.0 * omt * clamped * clamped + d = clamped * clamped * clamped + return ( + a * self.start[0] + b * self.control1[0] + c * self.control2[0] + d * self.end[0], + a * self.start[1] + b * self.control1[1] + c * self.control2[1] + d * self.end[1], + ) + + +@dataclass(frozen=True) +class MotionPlan: + """A fully resolved animation: where to draw, and for how long. + + This is the ONLY thing the supervisor sends to the overlay process, which is + why it is pre-sampled: the overlay is a dumb renderer with no Bezier in it, so + every decision about the shape of the motion stays in this testable module. + """ + + points: tuple[tuple[float, float], ...] + duration_ms: int + path: CursorPath + + +def curve_amount(distance: float, curve_scale: float = DEFAULT_CURVE_SCALE) -> float: + """The perpendicular arc magnitude for a move of *distance* pixels. + + ``clamp(distance * 0.22, 28, 110) * curve_scale`` — the reference's exact + formula, kept as three named constants in ``types``. The floor is what makes a + short nudge still read as a movement rather than a jump; the ceiling is what + stops a cross-screen sweep from bowing into a semicircle. + """ + scale = min(max(float(curve_scale), 0.0), MAX_CURVE_SCALE) + raw = max(float(distance), 0.0) * CURVE_DISTANCE_RATIO + return min(max(raw, CURVE_AMOUNT_MIN), CURVE_AMOUNT_MAX) * scale + + +def build_path( + start: tuple[float, float], + end: tuple[float, float], + *, + curve_scale: float = DEFAULT_CURVE_SCALE, + curve_direction: float = 0.0, +) -> CursorPath: + """Build the bowed cubic Bezier from *start* to *end*. + + *curve_direction* picks which side of the chord the bow falls on: positive is + the left-hand normal, negative the right-hand one, and ``0`` means "derive it + from the travel direction" (rightward moves bow one way, leftward the other, + so a there-and-back pair traces two different arcs instead of retracing one + line — the same trick the reference uses, and the reason repeated moves do not + look like a metronome). + + Degenerate input is handled rather than guarded against by the caller: a + zero-length move has no direction, so the normal would be ``0/0``. Distance is + floored at 1px and the direction falls back to +x, which yields a valid (if + pointless) path instead of a curve full of NaNs. That matters because NaN + coordinates reach AppKit as an un-placeable window rather than as an error. + """ + sx, sy = float(start[0]), float(start[1]) + ex, ey = float(end[0]), float(end[1]) + dx, dy = ex - sx, ey - sy + raw_distance = math.hypot(dx, dy) + distance = max(raw_distance, MIN_MOTION_DISTANCE) + + if raw_distance <= MOTION_EPSILON: + # No travel at all: there is no direction to derive a normal from (the + # naive normalize is 0/0), and more importantly there is nothing to + # animate. Collapse to a degenerate all-endpoints path so every sample is + # the same finite point — bowing a zero-length move would draw a pointless + # 28px loop around a stationary target. + return CursorPath( + start=(sx, sy), + end=(ex, ey), + control1=(sx, sy), + control2=(ex, ey), + curve_scale=0.0, + arc_amount=0.0, + ) + # Left-hand perpendicular of the unit travel vector. + norm_x, norm_y = -dy / raw_distance, dx / raw_distance + + direction = float(curve_direction) + if abs(direction) <= MOTION_EPSILON: + direction = 1.0 if dx >= 0.0 else -1.0 + else: + direction = 1.0 if direction > 0.0 else -1.0 + + scale = min(max(float(curve_scale), 0.0), MAX_CURVE_SCALE) + amount = curve_amount(distance, scale) + off_x = norm_x * amount * direction + off_y = norm_y * amount * direction + + if scale <= MOTION_EPSILON: + # curve_scale == 0: thirds placement with no offset is an exactly straight + # line. Kept as an explicit branch (rather than relying on amount == 0) + # so "no flourish" also means "no asymmetric easing in space". + c1 = (sx + dx * STRAIGHT_C1_FRACTION, sy + dy * STRAIGHT_C1_FRACTION) + c2 = (sx + dx * STRAIGHT_C2_FRACTION, sy + dy * STRAIGHT_C2_FRACTION) + return CursorPath( + start=(sx, sy), + end=(ex, ey), + control1=c1, + control2=c2, + curve_scale=0.0, + arc_amount=0.0, + ) + + c1 = (sx + dx * CURVE_C1_ALONG + off_x, sy + dy * CURVE_C1_ACROSS + off_y) + c2 = ( + sx + dx * CURVE_C2_ALONG + off_x * CURVE_C2_OFFSET_RATIO, + sy + dy * CURVE_C2_ACROSS + off_y * CURVE_C2_OFFSET_RATIO, + ) + return CursorPath( + start=(sx, sy), + end=(ex, ey), + control1=c1, + control2=c2, + curve_scale=scale, + arc_amount=amount, + ) + + +def spring_progress_curve(config: "SpringConfig | None" = None) -> tuple[float, ...]: + """Integrate the progress spring 0 -> 1 and return every sampled value. + + Velocity-Verlet at a FIXED ``dt``: the shape of the easing must not depend on + how fast the renderer happens to be running, so time is simulated here and the + renderer only decides how many of these samples it draws. + + The returned tuple always starts at exactly ``0.0`` and ends at exactly + ``1.0``. The final clamp is not cosmetic: the spring settles at + ``1.0000139`` for the shipped constants, and feeding a ``t > 1`` into a cubic + Bezier extrapolates past the target — the fake cursor would visibly shoot + past the element it is pointing at. Clamping progress (rather than clamping + the position afterwards) keeps the overshoot out of the model entirely. + + Bounded by ``SPRING_MAX_STEPS`` so a pathological configuration (a caller's + absurd ``response``, or a damping of 0 that never settles) terminates with a + usable-if-ugly curve instead of looping. + """ + cfg = config or SpringConfig.create() + values: list[float] = [0.0] + current = 0.0 + velocity = 0.0 + force = 0.0 + half_dt = cfg.dt * 0.5 + for _ in range(SPRING_MAX_STEPS): + velocity_half = velocity + force * half_dt + current = current + velocity_half * cfg.dt + force = cfg.stiffness * (1.0 - current) - cfg.drag * velocity_half + velocity = velocity_half + force * half_dt + if not math.isfinite(current): + # A caller-supplied configuration diverged. Bail with what we have; + # the sampler tolerates a short curve and the animation degrades to a + # quick move rather than raising into a tool call. + logger.debug("cursor-motion spring diverged; truncating progress curve") + break + values.append(current) + if current >= 1.0 and abs(1.0 - current) <= SPRING_SETTLE_DISTANCE: + break + values[-1] = 1.0 + return tuple(values) + + +def settle_time(config: "SpringConfig | None" = None) -> float: + """Simulated seconds for the progress spring to settle at 1.0. + + Derived from the same integration the easing uses, so the animation's + DURATION and its SHAPE can never drift apart — a duration picked + independently would either cut the settle off mid-ring-down or hold the + overlay after the cursor has stopped moving. Falls back to the measured + constant if the integrator bailed early. + """ + cfg = config or SpringConfig.create() + curve = spring_progress_curve(cfg) + steps = len(curve) - 1 + if steps <= 0: + return SPRING_FALLBACK_SETTLE_SECS + return steps * cfg.dt + + +def sample_path( + path: CursorPath, + *, + samples: int = DEFAULT_PATH_SAMPLES, + config: "SpringConfig | None" = None, +) -> tuple[tuple[float, float], ...]: + """Sample *path* at *samples* points, eased by the progress spring. + + The k-th output point is ``path.point_at(progress[k'])`` where ``k'`` walks the + spring's progress curve at uniform *time* intervals — so the points are + unevenly spaced in DISTANCE (dense at the start and end, sparse in the middle) + and evenly spaced in TIME. That is what produces ease-in/ease-out when a + renderer draws them at a constant frame rate, and it is why the renderer needs + no easing logic of its own. + + The first and last samples are the path's exact endpoints. The endpoint + guarantee is what the click pulse depends on: the pulse is drawn at the last + sampled point, so a sampler that stopped a pixel short would put the visual + click next to the element rather than on it. + """ + count = min(max(int(samples), MIN_PATH_SAMPLES), MAX_PATH_SAMPLES) + curve = spring_progress_curve(config) + last = len(curve) - 1 + out: list[tuple[float, float]] = [] + for step in range(count): + # Uniform in time across the spring's own step count, so a longer settle + # simply spreads the same number of drawn points over more of the curve. + idx = 0 if count == 1 else int(round(step * last / (count - 1))) + out.append(path.point_at(curve[min(idx, last)])) + out[0] = path.start + out[-1] = path.end + return tuple(out) + + +def plan_motion( + start: tuple[float, float], + end: tuple[float, float], + *, + curve_scale: float = DEFAULT_CURVE_SCALE, + curve_direction: float = 0.0, + samples: int = DEFAULT_PATH_SAMPLES, + config: "SpringConfig | None" = None, +) -> MotionPlan: + """Build the path, sample it, and clamp the duration — the one entry point. + + Duration comes from :func:`settle_time` (the spring's own settle point) and is + then clamped to ``[100, 2000]`` ms. The clamp is a product decision recorded + in ``types``: below 100ms the motion reads as a teleport and the affordance is + lost, and nothing purely cosmetic is allowed to hold a caller for longer than + 2s no matter what the spring says. + """ + cfg = config or SpringConfig.create() + distance = math.hypot(float(end[0]) - float(start[0]), float(end[1]) - float(start[1])) + + # A short hop is drawn STRAIGHT. ``curve_amount`` floors the arc at 28px, so a + # 1-3px nudge would otherwise bow ~28px out and back — a visible curlicue on a + # move the eye reads as "it barely moved". Verified before the change: a 1px + # move produced arc=28.0. + if distance < STRAIGHT_MOVE_DISTANCE: + curve_scale = 0.0 + path = build_path(start, end, curve_scale=curve_scale, curve_direction=curve_direction) + points = sample_path(path, samples=samples, config=cfg) + + # Scale the duration by distance. The spring's settle point is + # distance-INDEPENDENT, so using it raw gave a 1px nudge the same ~1429ms as a + # 600px sweep — which reads as a hang, not as motion. Above + # ``FULL_SPEED_DISTANCE`` the spring's own timing is used unchanged; below it + # the duration tapers linearly toward the floor. + raw_ms = settle_time(cfg) * 1000.0 + if distance < FULL_SPEED_DISTANCE: + raw_ms *= max(distance, 0.0) / FULL_SPEED_DISTANCE + duration_ms = min(max(int(round(raw_ms)), MIN_MOVE_DURATION_MS), MAX_MOVE_DURATION_MS) + return MotionPlan(points=points, duration_ms=duration_ms, path=path) diff --git a/src/kiro_crew/computer_use/enable_state.py b/src/kiro_crew/computer_use/enable_state.py new file mode 100644 index 00000000000..fc8daf6a440 --- /dev/null +++ b/src/kiro_crew/computer_use/enable_state.py @@ -0,0 +1,131 @@ +"""The KEYSTONE primary enable for computer use. + +State lives in ``/computer_use.json`` — **not** in ``config.json``. +That is a security decision with a precedent in this repo: the denied-command +opt-out is deliberately kept off ``config.json`` BECAUSE it is a security +ceiling, and ``security.py`` records the reasoning. A primary enable for full +desktop observation plus input synthesis is the same class of control, so it +gets the same treatment. + +The mechanics that make it un-flippable by the agent: + +* the leaf is on ``security._CREW_SECRET_LEAVES``, so ``is_sensitive_path`` + blocks agent reads AND writes on the tool path, and + ``is_sensitive_bash_command`` blocks the shell forms (``cat``, ``>``, + ``tar -x`` into the trust root); +* the only writer is the dashboard PUT handler, which does not route through the + agent tool gate; +* every read fails soft to ``{}`` → **DISABLED**. A missing, unreadable, + truncated or hand-mangled file must never mean "enabled" — fail-safe for a + gate whose open position hands out the operator's whole desktop. + +Modeled on ``hooks.load_denied_commands_state()``, which has the identical +contract and the identical failure posture. +""" + +from __future__ import annotations + +import json +import logging +from pathlib import Path +from typing import Any + +from kiro_crew.atomic_write import atomic_write +from kiro_crew.computer_use.types import ( + STATE_FILE_NAME, + STATE_KEY_ENABLED, + PolicyConfig, +) +from kiro_crew.config import loader as config_loader + +logger = logging.getLogger(__name__) + +# Owner-only: the file records a security decision, and its ``allowed_apps`` list +# names applications the operator automates — mildly sensitive on its own. +_STATE_FILE_MODE = 0o600 + + +def computer_use_state_path() -> Path: + """Path to the keystone ``computer_use.json``. + + Delegates to ``config.loader.computer_use_state_path()`` when that helper is + present so there is a single canonical definition (and so the suite's usual + ``patch("kiro_crew.config.loader.…")`` redirection keeps working). The + fallback computes the same path from ``config_dir()`` — resolved through the + loader module attribute, not a direct import, so a test that patches + ``config.loader.config_dir`` still redirects us. + """ + dedicated = getattr(config_loader, "computer_use_state_path", None) + if callable(dedicated): + path = dedicated() + return path if isinstance(path, Path) else Path(path) + return config_loader.config_dir() / STATE_FILE_NAME + + +def load_state() -> dict: + """Read the keystone state (fail-soft to ``{}``). + + Returns ``{}`` — which :func:`is_enabled` reads as DISABLED — when the file + is absent, unreadable, not valid JSON, or not a JSON object. A corrupt + ceiling file must not be interpreted generously. + """ + try: + raw = json.loads(computer_use_state_path().read_text(encoding="utf-8")) + return raw if isinstance(raw, dict) else {} + except FileNotFoundError: + return {} + except Exception: + logger.debug("computer_use.json load failed; treating as disabled", exc_info=True) + return {} + + +def is_enabled(state: "dict | None" = None) -> bool: + """True only when the keystone explicitly says ``enabled: true``. + + Strict identity against ``True`` rather than a truthiness test: a + hand-edited ``"enabled": "false"`` (a truthy non-empty string) or + ``"enabled": 1`` must not enable desktop control. The only spelling that + enables the feature is a real JSON ``true``, which is what the dashboard + writes. + + *state* lets a caller that already loaded the file avoid a second read + inside one dispatch. + """ + data = load_state() if state is None else state + return data.get(STATE_KEY_ENABLED) is True + + +def load_policy_config(state: "dict | None" = None) -> PolicyConfig: + """Operator target policy (allow-list / extra denials) from the keystone. + + Raises :class:`PolicyStateError` for ONE case: a present-but-malformed + ``allowed_apps``. Everything else is coerced, because an empty + ``extra_denied_apps`` only means "no extra denials" (fail-closed on its own), + while an empty ``allowed_apps`` means "every app that is not denied" — so + silently coercing a malformed allow-list would convert an operator's + restriction into no restriction at all. The dispatcher catches this and refuses + the action naming the file; refusing is the safe direction for a value that was + clearly INTENDED to narrow something. + + Note that an ABSENT ``allowed_apps`` is still the documented "no allow-list" + case and is not an error. + """ + return PolicyConfig.from_state(load_state() if state is None else state) + + +def save_state(state: dict) -> None: + """Write the keystone state atomically, owner-only. + + Callers must pass the COMPLETE object (read-modify-write): this replaces the + file wholesale, and the dashboard handler is responsible for refusing to + mutate a corrupt file rather than clobbering it — resetting a populated but + unparseable ceiling to defaults would be a silent security downgrade. + + Raises on failure (``OSError``) so the HTTP handler can report a real error; + the read path is the only fail-soft direction here. + """ + if not isinstance(state, dict): + raise ValueError("computer_use.json state must be a dict") + path = computer_use_state_path() + payload: dict[str, Any] = dict(state) + atomic_write(path, json.dumps(payload, indent=2) + "\n", mode=_STATE_FILE_MODE) diff --git a/src/kiro_crew/computer_use/gate.py b/src/kiro_crew/computer_use/gate.py new file mode 100644 index 00000000000..fe2232def68 --- /dev/null +++ b/src/kiro_crew/computer_use/gate.py @@ -0,0 +1,259 @@ +"""The computer-use enable check, and the audit of what it allowed. + +Deliberately small. An earlier revision of this module carried a full governance +model for computer use — eight ``SCOPE_CATALOG`` rows (capability, actions, apps, +app_names, observations, targets, approval, pointer), an unattended-surface +refusal, an interactive-approval floor, and a per-app disclosure filter. All of it +is gone: the product decision is that computer use is one operator opt-in, and +after that the agent drives the desktop the way the operator would. + +What remains here is the one question worth asking on every call — *is the feature +on?* — plus the SEL audit trail, so the operator has a record of what the agent did +to their desktop. + +Where the real protections live now, none of which is in this file: + +* the **primary enable** is on the keystone ``computer_use.json`` + (``enable_state``), which ``security._SENSITIVE_HOME_DIRS`` fences the agent away + from. That is what keeps "the agent cannot turn on its own desktop automation" + true, and it is checked at the dispatch chokepoint in :mod:`tools`; +* **KiroCrew's own window** is refused by :mod:`policy`, because driving our own + Settings UI would route around the keystone above; +* **password fields** are never read and never photographed (``ElementRec.secure`` + → the renderer emits a placeholder, and a window holding one gets no screenshot + at all). That is a privacy floor, not a policy knob; +* **credential redaction** on the way out is the repo-wide control every other + egress path already runs. + +Nothing here touches ctypes, the filesystem, or a platform framework, so this +module imports and tests identically on macOS, Linux, Windows and in CI. +""" + +from __future__ import annotations + +import logging +from typing import Any, Mapping + +from kiro_crew.computer_use.types import ( + ALL_OBSERVATION_CHANNELS, + AUDIT_POINTER_ITEM, + AUDIT_TOOL_PREFIX, + REFUSAL_POINTER_NOT_ENABLED, +) +from kiro_crew.platform.governance import ( + CU_CLASS_MUTATE, + computer_use_action_classes, +) + +logger = logging.getLogger(__name__) + +# Payload keys the renderers hand to :func:`apply_observation_ceiling`, and read +# back out of it. Named constants because ``tools`` and ``render`` both build and +# destructure these dicts, and a typo would silently drop a field rather than fail. +PAYLOAD_SCREENSHOT = "screenshot" +PAYLOAD_SCREENSHOT_META = ("screenshot_width", "screenshot_height", "screenshot_bytes") +PAYLOAD_WINDOW_TITLE = "window_title" +PAYLOAD_ELEMENTS = "elements" +PAYLOAD_APPS = "apps" +PAYLOAD_NOTES = "notes" +PAYLOAD_TEXT = "text" +ELEMENT_VALUE_KEY = "value" +ELEMENT_TITLE_KEY = "title" +APP_WINDOW_TITLE_KEY = "window_title" + + +def require_computer_use( + action: str, + *, + session_key: str, + agent: str = "", + app: str = "", + app_bundle_id: str = "", + app_display_name: str = "", + observations: "Any" = (), + target_roles: "Any" = (), + requires_app_identity: bool = True, + approval_recorded: bool = False, +) -> "str | None": + """Always returns ``None`` (proceed). Audits the call. + + The signature is preserved so ``tools.py``'s ordered chokepoint reads the same + and so a future edition can reintroduce a decision here without touching every + call site — but there is no longer a governance decision to make. The keystone + primary enable, checked upstream in :mod:`tools`, is the whole gate. + + Every parameter after ``action`` is accepted and ignored on purpose rather than + deleted: they are the identity the SEL audit records, and dropping them from the + signature would make the audit line poorer for no gain. + """ + _audit_allowed(session_key, agent, action, app_bundle_id or app_display_name) + return None + + +def require_pointer_move( + action: str, + *, + method: str, + session_key: str, + agent: str = "", + app: str = "", + pointer_enabled: bool = True, +) -> "str | None": + """Always returns ``None`` when the feature is on. + + The real-pointer path used to need a second opt-in (``allow_pointer_move``) and + a governance permit of its own. Both are gone: one enable covers the feature, + and ``policy.resolve_click_method`` still requires the model to NAME + ``click_method: "global"`` explicitly — ``auto`` never resolves to it, so the + pointer is never warped by accident. + + ``pointer_enabled`` is retained (defaulting to True) only so an in-process + caller can still refuse locally if it wants to. + """ + if not pointer_enabled: + return REFUSAL_POINTER_NOT_ENABLED + return None + + +def audit_pointer_move( + action: str, + *, + method: str, + session_key: str, + agent: str = "", + app_label: str = "", +) -> None: + """SEL-audit a real-pointer gesture. Best-effort; never raises. + + Retained after the governance removal because it is the one record that says + the operator's physical cursor was moved — the thing they would most want to + find in a log afterwards. + + ``tool_kind`` is deliberately its OWN value rather than the plain + ``computer_use`` every other call uses: it is what makes "did the agent ever + take control of my mouse?" answerable with one filter over the trail, instead of + requiring the reader to parse ``resources`` on every computer-use row. + """ + try: + from kiro_crew.sel import sel + + sel().log_tool_invocation( + session_key=session_key, + agent=agent or "kirocrew", + source="mcp", + tool_name=f"{AUDIT_TOOL_PREFIX}{action}", + tool_kind="computer_use_pointer", + outcome="ok", + resources=( + f"{AUDIT_POINTER_ITEM}={method}" + (f" app={app_label}" if app_label else "") + ), + ) + except Exception: + logger.debug("pointer-move audit failed", exc_info=True) + + +def is_mutating_action(action: str) -> bool: + """Whether *action* synthesizes input into another application. + + Reads the code-owned class table in ``platform/governance.py`` rather than a + private copy, so "which verbs are mutating" has one definition. + """ + return CU_CLASS_MUTATE in computer_use_action_classes(action) + + +def permitted_observation_channels( + *, session_key: str = "", agent: str = "", app: str = "" +) -> frozenset[str]: + """Every observation channel. No channel is withheld any more. + + Kept as a function (rather than inlining the constant at the call sites) so the + screenshot relay and the renderers keep one shared answer, and so an edition + that wants to narrow observations again has a single place to do it. + """ + return frozenset(ALL_OBSERVATION_CHANNELS) + + +def apply_observation_ceiling( + payload: Mapping[str, Any], *, session_key: str = "", agent: str = "", app: str = "" +) -> dict[str, Any]: + """Pass *payload* through unchanged. + + The ceiling used to blank window titles, strip element values and scrub file + paths according to the ``computer_use.observations`` scope. With that scope gone + there is nothing to narrow: the renderers' own secure-field suppression and the + package-wide credential redaction are what protect the output now. + """ + return dict(payload) + + +def targets_axis_is_governed(*, session_key: str = "", agent: str = "", app: str = "") -> bool: + """Always ``False`` — there is no ``targets`` ceiling to evaluate. + + ``tools`` reads this to decide whether to DEMAND an explicit ``element_index`` + for keyboard input and coordinate gestures. With the axis gone, both forms are + allowed: typing into whatever the app has focused is a legitimate flow again. + """ + return False + + +def app_is_disclosable( + *, + bundle_id: str, + display_name: str, + session_key: str = "", + agent: str = "", + app: str = "", +) -> bool: + """Whether ``computer_list_apps`` may name this application. + + Now only "does it have an identity at all" — an app with neither a bundle id + nor a display name is dropped because there is nothing to show, not because a + policy forbids it. The per-app governance axes that used to filter this list + are gone. + """ + return bool(bundle_id or display_name) + + +def _audit_allowed(session_key: str, agent: str, action: str, target: str) -> None: + """SEL-audit an allowed computer-use call. Best-effort; never raises. + + Every call is audited, not only the interesting ones: with the governance layer + removed, this trail is the operator's primary record of what the agent did to + their desktop. + """ + try: + from kiro_crew.sel import sel + + sel().log_tool_invocation( + session_key=session_key, + agent=agent or "kirocrew", + source="mcp", + tool_name=f"{AUDIT_TOOL_PREFIX}{action}", + tool_kind="computer_use", + outcome="ok", + resources=f"app={target}" if target else "", + ) + except Exception: + logger.debug("computer-use audit failed", exc_info=True) + + +__all__ = [ + "APP_WINDOW_TITLE_KEY", + "ELEMENT_TITLE_KEY", + "ELEMENT_VALUE_KEY", + "PAYLOAD_ELEMENTS", + "PAYLOAD_SCREENSHOT", + "PAYLOAD_SCREENSHOT_META", + "PAYLOAD_WINDOW_TITLE", + "PAYLOAD_APPS", + "PAYLOAD_NOTES", + "PAYLOAD_TEXT", + "app_is_disclosable", + "apply_observation_ceiling", + "audit_pointer_move", + "is_mutating_action", + "permitted_observation_channels", + "require_computer_use", + "require_pointer_move", + "targets_axis_is_governed", +] diff --git a/src/kiro_crew/computer_use/index.py b/src/kiro_crew/computer_use/index.py new file mode 100644 index 00000000000..53f0aedf333 --- /dev/null +++ b/src/kiro_crew/computer_use/index.py @@ -0,0 +1,296 @@ +"""Per-session, per-app snapshot cache — the element-index lifecycle. + +Element indices address a LIVE user interface, so the cache that maps an index +back to an element is a correctness control, not an optimization. Three +mechanisms bound its validity, none of which needs a turn-boundary signal +kiro-cli does not have: + +1. **Hard fail, never lazy re-snapshot.** Acting on an app with no cached + snapshot is refused. A lazy re-walk would silently let the model act on a + tree it was never shown, which is exactly the failure indices are supposed to + make impossible. +2. **TTL** (:data:`SNAPSHOT_TTL_SECS`). ``time.monotonic()`` throughout: a wall + clock adjustment must never make a stale snapshot look fresh. +3. **Fingerprint drift**, checked by the caller against a FRESH walk before + every mutating action. This module supplies the comparison; the driver + supplies the fresh tree. + +Plus :meth:`SnapshotIndex.end_turn` for an explicit early release. + +**Entries are keyed by ``(session_key, window_key)``** — session, then the specific +WINDOW, and both halves came from a review finding. + +The WINDOW half: element indices address one window's accessibility tree, so keying +by application alone aliased distinct windows of the same app. Snapshot document A, +focus document B, and the follow-up action — which re-resolves to B — retrieved A's +cached tree. The fingerprint check cannot catch it, because two documents of the same +app routinely have identically-shaped toolbars, so ``role|subrole|title`` at a given +index matches and the action mutates the wrong document. ``AppRef.window_key`` +carries the pid as well as the window id, since a window id is only unique within a +session and a relaunched app can reuse one. + +The SESSION half (reviewer finding). The gateway runs one +process serving every surface — dashboard tabs, Slack threads, cron jobs — so a +cache keyed by application ALONE is shared mutable state across concurrent +sessions: session A walks Preview and is shown indices, session B then walks +Preview after the UI moved and its snapshot REPLACES A's under the same key, and +A's next action resolves (and fingerprint-verifies) against B's tree. Both +sessions look internally consistent and the wrong control is activated, which is +the one outcome element addressing exists to prevent. The key is therefore +``(session_key, app_key)``, and lifecycle operations are per-session: one +session's ``computer_end_turn`` must not blow away another's live indices. + +The cap is per session as well, so a chatty session cannot evict another's +entries — the eviction path would otherwise be a cross-session denial of service +that surfaces as a confusing "call computer_get_state first". + +Honest limit, stated here and in the spec: fingerprinting narrows the race, it +does not eliminate it. A tree can still change between the verifying walk and +the action microseconds later. It converts a silent wrong-click into a loud +refusal in the overwhelming majority of cases; it is not a transactional +guarantee. Namespacing removes the CROSS-SESSION race entirely (two sessions can +no longer share an entry); it does not remove the within-session one, which is +inherent to driving a live UI. + +Pure: no ctypes, no I/O, no platform calls. +""" + +from __future__ import annotations + +import threading +import time + +from kiro_crew.computer_use.types import ( + ERR_INDEX_DRIFT, + ERR_NO_STATE, + ERR_STALE_STATE, + ERR_UNKNOWN_INDEX, + MAX_INDEXED_APPS, + SNAPSHOT_TTL_SECS, + TOOL_GET_STATE, + ElementRec, + Snapshot, + StaleIndex, +) + + +class SnapshotIndex: + """Bounded, TTL'd map of ``(session key, window key)`` -> last :class:`Snapshot`. + + Thread-safe: the MCP stdio loop dispatches tool calls on a worker thread + while the main thread reads stdin, and the gateway dispatches concurrent + sessions' calls on a thread pool — so every mutation is under one lock. + + Every accessor takes the ``session_key`` explicitly rather than reading an + ambient value. The dispatcher already carries that identity (it is the same + key the authoritative gate is queried with), and a thread-local would be + wrong here: the gateway runs each dispatch on a POOLED thread, so a + thread-local would attribute one session's snapshot to whichever session + reused the thread next. + """ + + def __init__( + self, + *, + ttl_secs: float = SNAPSHOT_TTL_SECS, + max_apps: int = MAX_INDEXED_APPS, + ) -> None: + self._ttl = ttl_secs + self._max_apps = max_apps + self._lock = threading.Lock() + # Insertion-ordered: dicts preserve order, so the oldest inserted key is + # the first one to evict once the cap is hit. Keyed by the COMPOSITE + # (session, app) — see the module docstring for why app alone is unsafe. + self._snapshots: dict[tuple[str, str], Snapshot] = {} + + def __len__(self) -> int: + with self._lock: + return len(self._snapshots) + + @property + def keys(self) -> tuple[tuple[str, str], ...]: + """Currently cached ``(session key, app key)`` pairs (diagnostics/tests).""" + with self._lock: + return tuple(self._snapshots) + + def app_keys(self, session_key: str) -> tuple[str, ...]: + """App keys cached for ONE session (diagnostics/tests).""" + with self._lock: + return tuple(app for sess, app in self._snapshots if sess == session_key) + + def _count_locked(self, session_key: str) -> int: + """Entries held by *session_key*. Caller holds the lock.""" + return sum(1 for sess, _app in self._snapshots if sess == session_key) + + def put(self, snap: Snapshot, *, session_key: str) -> None: + """Cache *snap* for *session_key*, replacing that session's prior entry. + + Evicts the oldest entry OF THIS SESSION when the per-session cap is hit. + Only a handful of apps are ever in play in one turn, so the cap exists to + bound RSS (each snapshot holds its encoded JPEG), not to manage + contention — and it is per-session so a chatty surface cannot evict + another's live indices. + """ + composite = (session_key, snap.key) + with self._lock: + self._snapshots.pop(composite, None) + self._snapshots[composite] = snap + while self._count_locked(session_key) > self._max_apps: + oldest = next(key for key in self._snapshots if key[0] == session_key) + self._snapshots.pop(oldest, None) + + def get(self, app_key: str, *, session_key: str, now: float | None = None) -> Snapshot | None: + """This session's cached snapshot for *app_key*, or ``None`` if absent/expired. + + An expired entry is DROPPED here rather than left to rot, so a later + call cannot resurrect it and the cap is not consumed by dead entries. + """ + stamp = time.monotonic() if now is None else now + composite = (session_key, app_key) + with self._lock: + snap = self._snapshots.get(composite) + if snap is None: + return None + if stamp - snap.captured_at > self._ttl: + self._snapshots.pop(composite, None) + return None + return snap + + def age(self, app_key: str, *, session_key: str, now: float | None = None) -> float: + """Age in seconds of this session's cached snapshot, or ``-1.0`` when absent. + + Read WITHOUT the TTL drop so an expiry message can quote the real age. + """ + stamp = time.monotonic() if now is None else now + with self._lock: + snap = self._snapshots.get((session_key, app_key)) + return -1.0 if snap is None else stamp - snap.captured_at + + def require( + self, + app_key: str, + app_label: str, + *, + session_key: str, + now: float | None = None, + ) -> Snapshot: + """Return a live snapshot for *app_key* or raise :class:`StaleIndex`. + + The two refusals are distinct on purpose: "call get_state first" and + "your state is N seconds old" tell the model different things, and the + age number is what makes the second one actionable. A snapshot another + session took is not visible here at all, so it produces the ordinary + "call get_state first" refusal rather than someone else's tree. + """ + stamp = time.monotonic() if now is None else now + age = self.age(app_key, session_key=session_key, now=stamp) + snap = self.get(app_key, session_key=session_key, now=stamp) + if snap is None: + if age < 0: + raise StaleIndex(ERR_NO_STATE.format(app=app_label, tool=TOOL_GET_STATE)) + raise StaleIndex( + ERR_STALE_STATE.format(app=app_label, age=int(age), tool=TOOL_GET_STATE) + ) + return snap + + def resolve(self, snap: Snapshot, element_index: int) -> ElementRec: + """Return the element at *element_index* within *snap*. + + Looked up by the record's own ``index`` field rather than by list + position: elided container nodes never consume an index, so position and + index are NOT interchangeable, and indexing by position would silently + address a different element than the one the model was shown. + """ + for rec in snap.elements: + if rec.index == element_index: + return rec + raise StaleIndex( + ERR_UNKNOWN_INDEX.format( + index=element_index, + app=snap.app.bundle_id or snap.app.name, + count=len(snap.elements), + ) + ) + + def invalidate(self, app_key: str, *, session_key: str) -> None: + """Drop THIS session's cached snapshot for one app (post-action, or on drift).""" + with self._lock: + self._snapshots.pop((session_key, app_key), None) + + def end_turn(self, *, session_key: str) -> None: + """Drop every snapshot cached for *session_key* — the explicit early release. + + Called by ``computer_end_turn``: once the model is done acting, holding + stale trees serves no purpose and every later index is a liability. + + Scoped to the calling session. A process-wide clear would let any surface + invalidate every OTHER surface's live indices, turning another session's + next action into a spurious "call computer_get_state first". + """ + with self._lock: + for key in [k for k in self._snapshots if k[0] == session_key]: + self._snapshots.pop(key, None) + + def clear(self) -> None: + """Drop EVERY session's snapshots — lifecycle only (backend reset/teardown). + + Deliberately not reachable from a tool: this is the process-wide reset the + backend swap needs (indices from one driver's walk are meaningless against + another's), whereas :meth:`end_turn` is the per-session release a model + can ask for. + """ + with self._lock: + self._snapshots.clear() + + +def drift_message( + app_label: str, + element_index: int, + before: str, + after: str, +) -> str: + """The refusal text for a fingerprint mismatch. + + Names BOTH the old and the new identity: without them the model cannot tell + whether the UI merely re-laid out or it was about to click something + genuinely different, and it would just retry blindly. + """ + return ERR_INDEX_DRIFT.format( + index=element_index, + tool=TOOL_GET_STATE, + before=before, + after=after, + ) + + +# ── Process-wide shared index ── +# The sidecar has exactly one snapshot cache: the dispatch chokepoint and the +# lifecycle hooks (``computer_end_turn``, backend reset) must all see the same +# map, and passing it through every call site would be threaded state for no +# benefit in a single-purpose process. + +_shared_index: SnapshotIndex | None = None +_shared_index_lock = threading.Lock() + + +def get_shared_index() -> SnapshotIndex: + """Process-wide snapshot cache singleton.""" + global _shared_index + with _shared_index_lock: + if _shared_index is None: + _shared_index = SnapshotIndex() + return _shared_index + + +def reset_shared_index() -> None: + """Drop the shared cache (tests, backend swap, KIROCREW_HOME changes). + + Swapping the backend MUST drop the cache: indices from one driver's walk are + meaningless against another's, and a fake backend inheriting a real + driver's snapshot (or the reverse) would be a genuine correctness bug. + """ + global _shared_index + with _shared_index_lock: + if _shared_index is not None: + _shared_index.clear() + _shared_index = None diff --git a/src/kiro_crew/computer_use/keymap.py b/src/kiro_crew/computer_use/keymap.py new file mode 100644 index 00000000000..81a7d1fcc4d --- /dev/null +++ b/src/kiro_crew/computer_use/keymap.py @@ -0,0 +1,188 @@ +"""US-layout keycode / modifier tables and the ``press_key`` spec parser. + +Pure data plus parsing — no ctypes, no platform calls, no I/O. The numbers are +Carbon virtual keycodes (``kVK_*`` from ``HIToolbox/Events.h``) and CoreGraphics +event-flag masks (``kCGEventFlagMask*``); both are stable ABI constants, which +is why they can live in a platform-free module and be unit-tested on Linux CI. + +Every synthesized key event must carry an EXPLICIT flag mask built from zero and +OR-ed with only the modifiers the caller asked for. Skipping that step made a +live prototype type ``' I Abc'`` when asked for ``abc``, because the events +inherited the user's real modifier state at post time. +""" + +from __future__ import annotations + +from kiro_crew.computer_use.types import KeyParseError + +# ── CoreGraphics event flag masks (kCGEventFlagMask*) ── +FLAG_ALPHA_SHIFT = 0x00010000 +FLAG_SHIFT = 0x00020000 +FLAG_CONTROL = 0x00040000 +FLAG_ALTERNATE = 0x00080000 +FLAG_COMMAND = 0x00100000 +FLAG_SECONDARY_FN = 0x00800000 + +# Modifier spellings a model might plausibly emit, all normalized to one mask. +# ``super``/``meta``/``win`` map to Command so a cross-platform prompt still +# works; ``fn`` is included because some app shortcuts require it. +MODIFIERS: dict[str, int] = { + "cmd": FLAG_COMMAND, + "command": FLAG_COMMAND, + "super": FLAG_COMMAND, + "meta": FLAG_COMMAND, + "win": FLAG_COMMAND, + "shift": FLAG_SHIFT, + "option": FLAG_ALTERNATE, + "opt": FLAG_ALTERNATE, + "alt": FLAG_ALTERNATE, + "control": FLAG_CONTROL, + "ctrl": FLAG_CONTROL, + "fn": FLAG_SECONDARY_FN, + "function": FLAG_SECONDARY_FN, + "capslock": FLAG_ALPHA_SHIFT, +} + +# ── Virtual keycodes (kVK_*), full US layout ── +# Keys are lowercase so lookup is case-insensitive after normalization. The +# named keys carry several aliases each (``esc``/``escape``, ``enter``/ +# ``return``, ``pgup``/``pageup``, …) because models are inconsistent and a +# ``KeyParseError`` for a spelling difference is a pointless failure. +KEYCODES: dict[str, int] = { + # letters + "a": 0, "b": 11, "c": 8, "d": 2, "e": 14, "f": 3, "g": 5, "h": 4, + "i": 34, "j": 38, "k": 40, "l": 37, "m": 46, "n": 45, "o": 31, "p": 35, + "q": 12, "r": 15, "s": 1, "t": 17, "u": 32, "v": 9, "w": 13, "x": 7, + "y": 16, "z": 6, + # digits + "0": 29, "1": 18, "2": 19, "3": 20, "4": 21, + "5": 23, "6": 22, "7": 26, "8": 28, "9": 25, + # punctuation (unshifted glyphs, plus a word alias for each) + "-": 27, "minus": 27, + "=": 24, "equal": 24, "equals": 24, + "[": 33, "leftbracket": 33, + "]": 30, "rightbracket": 30, + "\\": 42, "backslash": 42, + ";": 41, "semicolon": 41, + "'": 39, "quote": 39, "apostrophe": 39, + ",": 43, "comma": 43, + ".": 47, "period": 47, "dot": 47, + "/": 44, "slash": 44, + "`": 50, "grave": 50, "backtick": 50, + # whitespace / editing + "space": 49, " ": 49, "spacebar": 49, + "return": 36, "enter": 36, + "tab": 48, + "delete": 51, "backspace": 51, + "forwarddelete": 117, "del": 117, + "escape": 53, "esc": 53, + "help": 114, "insert": 114, + # navigation + "left": 123, "right": 124, "down": 125, "up": 126, + "arrowleft": 123, "arrowright": 124, "arrowdown": 125, "arrowup": 126, + "home": 115, "end": 119, + "pageup": 116, "pgup": 116, + "pagedown": 121, "pgdn": 121, "pgdown": 121, + # function keys + "f1": 122, "f2": 120, "f3": 99, "f4": 118, "f5": 96, "f6": 97, + "f7": 98, "f8": 100, "f9": 101, "f10": 109, "f11": 103, "f12": 111, + "f13": 105, "f14": 107, "f15": 113, "f16": 106, "f17": 64, "f18": 79, + "f19": 80, "f20": 90, + # keypad + "keypad0": 82, "keypad1": 83, "keypad2": 84, "keypad3": 85, "keypad4": 86, + "keypad5": 87, "keypad6": 88, "keypad7": 89, "keypad8": 91, "keypad9": 92, + "keypadclear": 71, "keypaddecimal": 65, "keypaddivide": 75, + "keypadenter": 76, "keypadequals": 81, "keypadminus": 78, + "keypadmultiply": 67, "keypadplus": 69, + # media / volume (bare keycodes; no special HID handling needed) + "mute": 74, "volumedown": 73, "volumeup": 72, +} # fmt: skip + +# Characters reachable only with Shift on a US layout. Used when text has to be +# typed as keystrokes (no addressable element to set a value on): the keystroke +# for ``$`` is Shift+4, and the shift flag must be applied to that event only. +SHIFTED_CHARS: dict[str, str] = { + "~": "`", "!": "1", "@": "2", "#": "3", "$": "4", "%": "5", + "^": "6", "&": "7", "*": "8", "(": "9", ")": "0", "_": "-", + "+": "=", "{": "[", "}": "]", "|": "\\", ":": ";", '"': "'", + "<": ",", ">": ".", "?": "/", +} # fmt: skip + +# Separators accepted between modifiers and the key in a spec string. ``-`` is +# NOT one of them: it is itself a key name, so ``cmd-`` would be ambiguous. +_SPEC_SEPARATOR = "+" + + +def parse_key(spec: str) -> tuple[int, int]: + """Parse a key spec like ``"cmd+shift+a"`` into ``(keycode, flag_mask)``. + + The mask is built from ZERO — never from the current modifier state — so a + synthesized event carries exactly the modifiers that were requested. + + Raises :class:`KeyParseError` for an empty spec, an unknown modifier, an + unknown key, or a spec with no key part (``"cmd+"``). Refusing loudly is + correct here: silently dropping an unrecognized modifier would send a + DIFFERENT keystroke than the caller asked for, into a live application. + """ + if not isinstance(spec, str) or not spec.strip(): + raise KeyParseError("empty key spec") + raw = spec.strip() + + # A bare ``+`` (or a spec ending in one, e.g. ``shift++``) means the plus + # key itself, which is Shift+equal on a US layout. Handle it before + # splitting, or the split would yield an empty key part. + parts: list[str] = [] + if raw == _SPEC_SEPARATOR: + parts = ["+"] + elif raw.endswith(_SPEC_SEPARATOR): + parts = [p for p in raw[:-1].split(_SPEC_SEPARATOR) if p] + ["+"] + else: + parts = raw.split(_SPEC_SEPARATOR) + + tokens = [p.strip() for p in parts if p.strip()] + if not tokens: + raise KeyParseError(f"no key in spec {spec!r}") + + flags = 0 + for token in tokens[:-1]: + mask = MODIFIERS.get(token.lower()) + if mask is None: + raise KeyParseError(f"unknown modifier {token!r} in {spec!r}") + flags |= mask + + key = tokens[-1] + keycode, extra = _resolve_key(key) + if keycode is None: + raise KeyParseError(f"unknown key {key!r} in {spec!r}") + return keycode, flags | extra + + +def char_keystroke(char: str) -> tuple[int, int] | None: + """Return ``(keycode, flag_mask)`` for a single printable character. + + For the keystroke-synthesis path (typing text into a target that exposes no + settable value). Returns ``None`` for a character the US layout cannot + reach with one keystroke — callers must skip it rather than substitute + something else, since typing the wrong character into a live app is worse + than typing nothing. + """ + if not char: + return None + keycode, flags = _resolve_key(char) + return None if keycode is None else (keycode, flags) + + +def _resolve_key(key: str) -> tuple[int | None, int]: + """Resolve one key token to ``(keycode | None, implied_flags)``. + + Implied flags cover the shifted glyphs (``$`` -> Shift+4) and uppercase + letters (``A`` -> Shift+a); a caller-supplied ``shift+`` simply ORs into the + same bit, so both spellings produce an identical event. + """ + if not key: + return None, 0 + if key in SHIFTED_CHARS: + return KEYCODES.get(SHIFTED_CHARS[key]), FLAG_SHIFT + if len(key) == 1 and key.isalpha() and key.isupper(): + return KEYCODES.get(key.lower()), FLAG_SHIFT + return KEYCODES.get(key.lower()), 0 diff --git a/src/kiro_crew/computer_use/linux_driver.py b/src/kiro_crew/computer_use/linux_driver.py new file mode 100644 index 00000000000..81d97bc0cfc --- /dev/null +++ b/src/kiro_crew/computer_use/linux_driver.py @@ -0,0 +1,49 @@ +"""Linux computer-use backend — a typed refusal until a driver exists. + +Subclasses :class:`UnsupportedBackend`, so every tool answers with the same clear +"not supported on this platform" result instead of raising. + +**Imports nothing native.** No D-Bus, no GI bindings — importing this module is +stdlib-only on every platform, which keeps the package import-safe on the CI +fleet and lets a test exercise this path by flipping +``platform_compat.IS_LINUX``. + +Implementation plan for whoever writes the real driver: + +* **Tree** — AT-SPI 2 over D-Bus (``org.a11y.atspi.Accessible``), reached from + the session bus address in ``org.a11y.Bus``. Requires the toolkit-side + accessibility bridge to be enabled (``GTK_MODULES=gail:atk-bridge`` / + ``QT_ACCESSIBILITY=1``), which is NOT the default on every desktop — so the + driver must detect a missing bridge and report it as the reason rather than + returning an empty tree that looks like a working app with no controls. + ``ATSPI_STATE_PROTECTED`` (Qt/GTK password entries) is the + ``AXSecureTextField`` analogue and MUST drive ``ElementRec.secure``. +* **Capture is the hard part, and it is a genuine design fork.** On X11, + ``XGetImage`` over the window works directly. On **Wayland there is no + client-side screen capture at all**: a compositor screenshot requires + ``xdg-desktop-portal``'s ``org.freedesktop.portal.Screenshot``, which shows an + interactive consent dialog — unusable from a background sidecar. So a first + Linux cut is very likely **tree-only**. That is already accommodated: the + contract lets ``snapshot()`` return a tree with no image, ``want_image`` is a + request rather than a requirement, and the renderer simply omits the + screenshot line. +* **Input** — ``XTestFakeKeyEvent`` on X11 (global, not per-window). Wayland has + no equivalent for an unprivileged client; ``libei``/``xdg-desktop-portal`` + ``RemoteDesktop`` is the forward path and again requires consent. Decide the + focus-stealing question explicitly before shipping input, exactly as on + Windows — do not quietly steal focus. +* **App list** — enumerate AT-SPI applications and their frames; take the pid + from the application object, never from a process-name search. +""" + +from __future__ import annotations + +from kiro_crew.computer_use.backend import LINUX_REASON, UnsupportedBackend +from kiro_crew.computer_use.types import PLATFORM_LINUX + + +class LinuxBackend(UnsupportedBackend): + """Linux placeholder backend: reports unsupported, refuses every action.""" + + def __init__(self) -> None: + super().__init__(PLATFORM_LINUX, LINUX_REASON) diff --git a/src/kiro_crew/computer_use/macos_driver.py b/src/kiro_crew/computer_use/macos_driver.py new file mode 100644 index 00000000000..1fe83b7ee96 --- /dev/null +++ b/src/kiro_crew/computer_use/macos_driver.py @@ -0,0 +1,821 @@ +"""The macOS :class:`ComputerUseBackend` — thin glue over the Stage-2 modules. + +Deliberately thin. Discovery lives in :mod:`apps_macos`, the walk in +:mod:`snapshot_macos`, capture in :mod:`capture_macos`, the FFI in +:mod:`macos_ffi`, and every *policy* decision lives upstream at the dispatch +chokepoint. This class only translates the ABC's method signatures into those +calls and, critically, converts every failure into a :class:`DriverResult`. + +**No exception crosses this seam.** Each public method wraps its work in +:func:`_guarded`. That is not defensive habit: the MCP loop dispatches tool calls +on a worker thread, and an exception escaping into it takes the call down, while an +un-caught ``ComputerUseUnsupported`` at import time would make the whole backend +look broken rather than the one action. + +Two design notes that a reader will otherwise question: + +* **``element`` handles are not cached across calls.** A mutating action re-walks + the tree to find the element at the requested index. Holding a + ``AXUIElement`` pointer between tool calls would mean acting on a stale handle + whose widget may have been destroyed — a use-after-free in another process's + address space, not a Python error. Re-walking costs 25-70ms and is what makes + the caller's fingerprint check meaningful, since the fingerprint must come from + the same walk as the element being acted on. +* **Three click paths, and only one of them touches the operator's cursor.** + ``accessibility`` is ``AXUIElementPerformAction(elem, "AXPress")`` — no pointer + involved at all, and still the default whenever an element index is available. + ``app_post`` posts a located mouse event with ``CGEventPostToPid``, so the target + app sees a click at a point while the physical cursor stays put (verified live). + ``global`` is the one path that warps the real cursor + (``CGWarpMouseCursorPosition`` + a global ``CGEventPost``); it exists because + some UI is only reachable by a physical click, and it is reachable only when the + model NAMES it — ``policy.resolve_click_method`` never resolves ``auto`` onto it. + This driver does not evaluate that; it is settled at the dispatch chokepoint + before ``req.moves_pointer`` can be True here. A driver must therefore never + upgrade a method on its own. +""" + +from __future__ import annotations + +import logging +from contextlib import contextmanager +from typing import Callable, Iterator + +from kiro_crew.computer_use import ( + apps_macos, + capture_macos, + keymap, + macos_ffi, + macos_skylight, + permissions, + snapshot_macos, +) +from kiro_crew.computer_use.backend import ComputerUseBackend +from kiro_crew.computer_use.types import ( + AX_MENU_LADDER, + AX_PRESS_ACTION, + AX_PRESS_LADDER, + AX_VALUE, + CLICK_METHOD_ACCESSIBILITY, + CLICK_METHOD_APP_POST, + CLICK_METHOD_GLOBAL, + CLICK_METHOD_SKY_CLICK, + ERR_ACTION_FAILED, + ERR_POINT_NOT_OWNED, + ERR_POINT_REQUIRED, + ERR_UNKNOWN_CLICK_METHOD, + MAX_ANCESTOR_AREA_RATIO, + MAX_ANCESTOR_PRESS_HOPS, + MOUSE_BUTTON_LEFT, + MOUSE_BUTTON_RIGHT, + PERMISSION_UNKNOWN, + PLATFORM_MACOS, + REFUSAL_ACCESSIBILITY_NEEDS_INDEX, + SCROLL_DOWN, + SCROLL_LEFT, + SCROLL_RIGHT, + SCROLL_UP, + AppRef, + BackendStatus, + ClickRequest, + ComputerUseError, + DragRequest, + DriverResult, + ElementRec, + PermissionProbe, + SnapshotRequest, +) + +# The AX attribute set so typed keystrokes land in the addressed element rather +# than wherever the target app last had focus. +AX_FOCUSED = "AXFocused" + +logger = logging.getLogger(__name__) + +# The accessibility action tried for each scroll direction BEFORE falling back to +# a synthesized wheel event. Tried first because an AX scroll is scoped to the +# addressed element, while a wheel event goes to whatever the target app decides is +# under its scroll focus. +_SCROLL_ACTIONS: dict[str, str] = { + SCROLL_UP: "AXScrollUpByPage", + SCROLL_DOWN: "AXScrollDownByPage", + SCROLL_LEFT: "AXScrollLeftByPage", + SCROLL_RIGHT: "AXScrollRightByPage", +} +# Wheel deltas per page for each direction. macOS wheel deltas are inverted +# relative to the reading direction: a NEGATIVE axis-1 delta scrolls the content +# DOWN (the gesture of pushing the wheel away from you). +_SCROLL_DELTAS: dict[str, tuple[int, int]] = { + SCROLL_UP: (1, 0), + SCROLL_DOWN: (-1, 0), + SCROLL_LEFT: (0, 1), + SCROLL_RIGHT: (0, -1), +} +# Wheel-event page deltas are integers, so a fractional ``pages`` request is +# rounded up to at least one page — asking to scroll and being told "ok" while +# nothing moved is worse than over-scrolling by a fraction. +_MIN_WHEEL_PAGES = 1 + +_REASON_UNSUPPORTED_TARGET = "element advertises no {action} action and none succeeded" +_REASON_NO_ELEMENT = "element {index} is no longer present in '{app}'" +_REASON_AX_ERROR = "accessibility error {code}" +# Keyboard input whose addressed element could not be focused. ACTIONABLE on purpose +# (it names the fix) because the alternative — typing into whatever the app happened +# to have focused — is how a keystroke reaches a password field that +# ``policy.check_input_target`` already refused. +_REASON_FOCUS_FAILED = ( + "could not focus element {index} of '{app}' (accessibility error {code}), so the " + "keystrokes were NOT sent — they would have gone to whatever that app currently " + "has focused, which is not the target you named. Call computer_get_state again " + "and address a focusable control, or click it first" +) +# ``sky_click`` routes by window id, so a vanished window is a distinct failure from +# a vanished element: the model must re-snapshot to get a live id, and telling it +# that is what stops it retrying the same dead id. +_REASON_SKY_WINDOW_GONE = ( + "the target window is no longer on screen (or no longer belongs to that app), so " + "click_method 'sky_click' has no window to address. Call computer_get_state again" +) + + +class MacOSBackend(ComputerUseBackend): + """Accessibility + CoreGraphics driver for macOS. + + Constructing this class does NOT load a framework: the FFI layer binds lazily + on first real use, so a ``MacOSBackend()`` on a machine where + ApplicationServices will not load still constructs, and the first call returns + a typed refusal instead of exploding at import time. + """ + + def __init__(self) -> None: + # Cached because ``status()`` is called for every dashboard render and each + # miss would otherwise re-run ``find_library`` five times. + self._available: "bool | None" = None + + @property + def platform_id(self) -> str: + return PLATFORM_MACOS + + def status(self) -> BackendStatus: + """Whether the native frameworks load here.""" + if self._available is None: + self._available = macos_ffi.available() + if self._available: + return BackendStatus(supported=True, platform_id=PLATFORM_MACOS, reason="") + return BackendStatus( + supported=False, + platform_id=PLATFORM_MACOS, + reason=( + "the macOS accessibility frameworks could not be loaded " + "(ApplicationServices / CoreGraphics / ImageIO)" + ), + ) + + def probe_permissions(self) -> PermissionProbe: + """ADVISORY permission hints — never a gate. See :mod:`permissions`.""" + try: + return permissions.probe() + except Exception: + logger.debug("permission probe failed", exc_info=True) + return PermissionProbe( + accessibility=PERMISSION_UNKNOWN, screen_recording=PERMISSION_UNKNOWN + ) + + # ── observation ── + + def list_apps(self) -> DriverResult: + def run() -> DriverResult: + return DriverResult(ok=True, apps=apps_macos.list_apps()) + + return _guarded("list_apps", run) + + def resolve_app(self, query: str) -> DriverResult: + def run() -> DriverResult: + return DriverResult(ok=True, app=apps_macos.resolve_app(query)) + + return _guarded("resolve_app", run) + + def snapshot(self, app: AppRef, req: SnapshotRequest) -> DriverResult: + def run() -> DriverResult: + snap = snapshot_macos.build_snapshot(app, req) + if req.want_image: + # Capture is a separate step so the secure-field suppression rule + # lives in exactly one place (``capture_snapshot_image`` refuses on + # ``has_secure``) instead of being an argument threaded into the + # walk. A capture failure degrades to a tree-only snapshot. + snap = capture_macos.capture_snapshot_image( + snap, max_px=req.image_max_px, quality=req.image_quality + ) + return DriverResult(ok=True, app=app, snapshot=snap) + + return _guarded("snapshot", run) + + # ── input ── + + def click( + self, + app: AppRef, + rec: "ElementRec | None", + req: ClickRequest, + ) -> DriverResult: + """Dispatch one click onto the method *req* already resolved to. + + ``req.method`` is CONCRETE here — ``auto`` was resolved at the dispatch + chokepoint (``policy.resolve_click_method``), so this method never chooses + one and in particular can never select the pointer-warping path on its own. + An unrecognised method is REFUSED rather than falling back to + accessibility: a fallback would silently perform a different gesture than + the caller asked for. + """ + + def run() -> DriverResult: + if req.method == CLICK_METHOD_ACCESSIBILITY: + return self._click_accessibility(app, rec, req.button) + if req.point is None: + return DriverResult(ok=False, text=ERR_POINT_REQUIRED.format(method=req.method)) + if req.method == CLICK_METHOD_APP_POST: + # App-scoped: the target app sees a click at the point and the + # operator's physical cursor does NOT move. + macos_ffi.post_mouse_click( + app.pid, + req.point[0], + req.point[1], + button=req.button, + count=req.count, + ) + return DriverResult(ok=True, text=_click_text(req, app), app=app) + if req.method == CLICK_METHOD_SKY_CLICK: + # The PRIVATE background-window path. Reached only when the model + # NAMED it (``auto`` never resolves here), and implemented entirely + # in the quarantined ``macos_skylight`` module. + # + # No ``pid_owns_point`` check, and that is correct rather than an + # omission: this path carries the TARGET pid and window id in the + # event itself, so the window server routes to that window whatever + # is on top. The confinement ``global`` needs — "the pixel must + # belong to the authorized app" — exists because a global event has + # no pid to route by. Here the routing IS the confinement, and + # ``policy.check_app`` already authorized this pid. + bounds = apps_macos.window_bounds(app.window_id, app.pid) + if bounds is None: + return DriverResult(ok=False, text=_REASON_SKY_WINDOW_GONE) + left, top, width, height = bounds + return _sky_click(app, req, left, top, width, height) + if req.method == CLICK_METHOD_GLOBAL: + # THE pointer-moving path. Both permits were checked upstream; this + # driver must not re-derive or relax them. + # + # It MUST, however, confine the click to the app those permits were + # granted for. A global event carries no pid and lands on whatever + # owns the pixel, so naming an allowed app while passing coordinates + # over a denied one (a terminal, a password manager) would pass the + # policy check on app A and click app B. Every other input path is + # app-scoped and needs no such check. + if not apps_macos.pid_owns_point(app.pid, req.point[0], req.point[1]): + return DriverResult( + ok=False, + text=ERR_POINT_NOT_OWNED.format( + app=app.name, x=int(req.point[0]), y=int(req.point[1]) + ), + ) + macos_ffi.post_mouse_global( + req.point[0], + req.point[1], + button=req.button, + count=req.count, + ) + return DriverResult(ok=True, text=_click_text(req, app), app=app) + return DriverResult(ok=False, text=ERR_UNKNOWN_CLICK_METHOD.format(method=req.method)) + + return _guarded("click", run) + + def drag(self, app: AppRef, req: DragRequest) -> DriverResult: + """Drag between two screen points, app-scoped unless *req* moves the pointer.""" + + def run() -> DriverResult: + if req.method == CLICK_METHOD_GLOBAL: + # BOTH endpoints, for the same reason as the click above: a sweep + # that starts inside the authorized window and ends over a denied + # app would otherwise release the button there. + for point in (req.start, req.end): + if not apps_macos.pid_owns_point(app.pid, point[0], point[1]): + return DriverResult( + ok=False, + text=ERR_POINT_NOT_OWNED.format( + app=app.name, x=int(point[0]), y=int(point[1]) + ), + ) + macos_ffi.post_mouse_drag_global(req.start, req.end, button=req.button) + elif req.method == CLICK_METHOD_APP_POST: + macos_ffi.post_mouse_drag(app.pid, req.start, req.end, button=req.button) + else: + # ``accessibility`` has no drag form at all — no AX action expresses + # a sweep between two points — so an unusable method is refused + # rather than approximated. + return DriverResult( + ok=False, text=ERR_UNKNOWN_CLICK_METHOD.format(method=req.method) + ) + return DriverResult( + ok=True, + text=( + f"dragged with the {req.button} button from " + f"({req.start[0]:.0f}, {req.start[1]:.0f}) to " + f"({req.end[0]:.0f}, {req.end[1]:.0f}) in '{app.name}' " + f"({req.method})" + ), + app=app, + ) + + return _guarded("drag", run) + + def _click_accessibility( + self, app: AppRef, rec: "ElementRec | None", button: str = MOUSE_BUTTON_LEFT + ) -> DriverResult: + """Activate the addressed element through accessibility — no pointer at all. + + Tries a LADDER of actions rather than ``AXPress`` alone. A single press was + the biggest avoidable refusal in the driver: ``AXPress`` returns + ``-25206`` (action unsupported) on a disclosure triangle, a Finder row and + many web controls, all of which answer ``AXConfirm`` or ``AXOpen`` — so the + model was told "the click failed" for elements that were perfectly + clickable, and its only remaining move was to guess coordinates. + + A RIGHT button takes a different ladder (``AXShowMenu`` only) and never + falls back to a press: quietly turning "open the context menu" into + "activate the control" would perform a different action than the model + asked for, which is the same class of defect as + :func:`keymap.parse_key` refusing an unknown modifier. + + The winning action is NAMED in the result, so a model that sees ``AXOpen`` + learns what this element responds to instead of re-deriving it next turn. + """ + if rec is None: + return DriverResult(ok=False, text=REFUSAL_ACCESSIBILITY_NEEDS_INDEX) + ladder = AX_MENU_LADDER if button == MOUSE_BUTTON_RIGHT else AX_PRESS_LADDER + with _addressed(app, rec) as element: + if element is None: + return _missing_element(app, rec) + # ``advertised`` is what the element SAYS it supports. Preferred over + # blind attempts so an unsupported action is skipped without a + # round-trip — but an advertised action can still fail, so the result + # code is what decides, not the advertisement. + advertised = set(rec.actions or ()) + last_action, last_err = ladder[0], macos_ffi.AX_ERR_ACTION_UNSUPPORTED + for action in ladder: + if advertised and action not in advertised: + continue + err = macos_ffi.ax_perform(element, action) + if err == macos_ffi.AX_ERR_SUCCESS: + return DriverResult(ok=True, text=_pressed_text(rec.index, action), app=app) + last_action, last_err = action, err + if err != macos_ffi.AX_ERR_ACTION_UNSUPPORTED: + # A REAL failure (a wedged app, a destroyed widget), not "wrong + # verb" — trying the next verb would report the wrong cause. + break + else: + # Nothing in the ladder was advertised at all: attempt the primary + # verb anyway. Advertisement is unreliable on web content, where a + # node can perform a press it never listed. + if not any(a in advertised for a in ladder): + last_err = macos_ffi.ax_perform(element, ladder[0]) + last_action = ladder[0] + if last_err == macos_ffi.AX_ERR_SUCCESS: + return DriverResult( + ok=True, text=_pressed_text(rec.index, last_action), app=app + ) + # LAST rung: the element itself is unpressable, so try the container + # that visually IS it. Web content renders a clickable row as a plain + # text node inside a pressable ancestor, which leaves the whole row + # dead to an element click while a coordinate click on it works fine — + # and coordinates are exactly what this path exists to avoid. + # + # Only for a LEFT click, and only when the element carries a frame to + # compare against: without geometry there is no way to tell the row + # from the page, and pressing the page would activate something the + # model never addressed. See :func:`_ancestor_press`. + if button != MOUSE_BUTTON_RIGHT and last_err in ( + macos_ffi.AX_ERR_ACTION_UNSUPPORTED, + macos_ffi.AX_ERR_ATTRIBUTE_UNSUPPORTED, + ): + pressed = _ancestor_press(element, rec) + if pressed: + return DriverResult(ok=True, text=_ancestor_pressed_text(rec.index), app=app) + return _action_failed(app, rec, last_action, last_err) + + def type_text(self, app: AppRef, rec: "ElementRec | None", text: str) -> DriverResult: + def run() -> DriverResult: + if rec is not None: + with _addressed(app, rec) as element: + if element is None: + return _missing_element(app, rec) + # Focus the addressed element first, so the keystrokes land + # where the model aimed rather than wherever the target app + # last had focus. + focus_err = macos_ffi.ax_set_true(element, AX_FOCUSED) + # REFUSE on a focus failure — do not fall through to app-scoped + # input. ``policy.check_input_target`` cleared the element the model + # ADDRESSED (that is where ``ElementRec.secure`` comes from), so if + # focus did not move, ``post_text`` delivers to whatever the app + # already had focused — which can be the password field the check + # just refused to write into. Typing blind past a failed aim is the + # one case where "best effort" defeats the secure-field floor. + if focus_err != macos_ffi.AX_ERR_SUCCESS: + return _focus_failed(app, rec, focus_err) + # Unicode key events rather than a per-character keycode lookup: this + # is layout-independent and can emit characters the US layout cannot + # reach in one keystroke. Verified byte-identical round-trip. + macos_ffi.post_text(app.pid, text) + target = "the focused element" if rec is None else f"element {rec.index}" + return DriverResult( + ok=True, text=f"typed {len(text)} character(s) into {target}", app=app + ) + + return _guarded("type_text", run) + + def press_key(self, app: AppRef, rec: "ElementRec | None", key: str) -> DriverResult: + def run() -> DriverResult: + # ``parse_key`` raises KeyParseError (a ComputerUseError) for an + # unknown key or modifier, which ``_guarded`` turns into a refusal. + # Refusing loudly is right: silently dropping an unrecognised modifier + # would send a DIFFERENT keystroke than requested into a live app. + keycode, flags = keymap.parse_key(key) + if rec is not None: + # Focus the addressed element first, same as ``type_text``, and + # best-effort for the same reason. + with _addressed(app, rec) as element: + if element is None: + return _missing_element(app, rec) + focus_err = macos_ffi.ax_set_true(element, AX_FOCUSED) + # Same hole, same fix, the other keyboard verb — a keystroke + # delivered to a focus the model did not address can reach the + # secure field the policy check just cleared us away from. + if focus_err != macos_ffi.AX_ERR_SUCCESS: + return _focus_failed(app, rec, focus_err) + macos_ffi.post_key(app.pid, keycode, flags) + return DriverResult(ok=True, text=f"sent {key}", app=app) + + return _guarded("press_key", run) + + def set_value(self, app: AppRef, rec: ElementRec, value: str) -> DriverResult: + def run() -> DriverResult: + with _addressed(app, rec) as element: + if element is None: + return _missing_element(app, rec) + err = macos_ffi.ax_set_str(element, AX_VALUE, value) + if err != macos_ffi.AX_ERR_SUCCESS: + return _action_failed(app, rec, f"set {AX_VALUE}", err) + return DriverResult(ok=True, text=f"set element {rec.index}", app=app) + + return _guarded("set_value", run) + + def scroll(self, app: AppRef, rec: ElementRec, direction: str, pages: float) -> DriverResult: + def run() -> DriverResult: + action = _SCROLL_ACTIONS.get(direction) + if action is None: + return DriverResult(ok=False, text=f"unknown scroll direction '{direction}'") + whole_pages = max(_MIN_WHEEL_PAGES, int(round(pages))) + with _addressed(app, rec) as element: + if element is None: + return _missing_element(app, rec) + # Try the AX action first — it is element-scoped. But NEVER trust + # the advertised action list: a real AXScrollArea listed + # ``AXScrollDownByPage`` and then returned + # ``kAXErrorActionUnsupported`` when asked to perform it. So the + # advertised list is not even consulted; we just try and check. + err = macos_ffi.AX_ERR_SUCCESS + for _ in range(whole_pages): + err = macos_ffi.ax_perform(element, action) + if err != macos_ffi.AX_ERR_SUCCESS: + break + if err == macos_ffi.AX_ERR_SUCCESS: + return DriverResult( + ok=True, + text=f"scrolled element {rec.index} {direction} {whole_pages} page(s)", + app=app, + ) + + # Fall back to a synthesized wheel event, posted to the target pid with + # the private event source. + # The wheel event is in LINE units — ``CGScrollEventUnit`` has only + # ``Pixel`` and ``Line``, no page unit — so a requested PAGE has to be + # expressed as a line count. Sending the raw page number scrolled one + # single line and reported "scrolled 1 page(s)", which is the kind of + # silent no-op that makes an agent loop. + delta_y, delta_x = _SCROLL_DELTAS[direction] + lines = whole_pages * macos_ffi.CG_SCROLL_LINES_PER_PAGE + macos_ffi.post_scroll(app.pid, delta_y * lines, delta_x * lines) + return DriverResult( + ok=True, + text=( + f"scrolled '{app.name}' {direction} {whole_pages} page(s) with a wheel " + f"event (the accessibility action was refused with error {err})" + ), + app=app, + ) + + return _guarded("scroll", run) + + def perform_action(self, app: AppRef, rec: ElementRec, action: str) -> DriverResult: + def run() -> DriverResult: + with _addressed(app, rec) as element: + if element is None: + return _missing_element(app, rec) + err = macos_ffi.ax_perform(element, action) + if err != macos_ffi.AX_ERR_SUCCESS: + return _action_failed(app, rec, action, err) + return DriverResult(ok=True, text=f"performed {action} on element {rec.index}", app=app) + + return _guarded("perform_action", run) + + def close(self) -> None: + """Release the FFI handles, the event source and the identity cache. + + Safe to call repeatedly. Called by ``reset_shared_backend``, which is also + what drops the snapshot index — element indices from one driver's walk are + meaningless against another's. + """ + try: + apps_macos.reset_identity_cache() + except Exception: + logger.debug("identity cache reset failed", exc_info=True) + try: + macos_ffi.reset_frameworks() + except Exception: + logger.debug("framework reset failed", exc_info=True) + + +@contextmanager +def _addressed(app: AppRef, rec: ElementRec) -> "Iterator[object | None]": + """Yield the live element at ``rec.index``, or ``None``; always releases it. + + Re-resolves on every call rather than caching a handle across tool calls: a + stored ``AXUIElement`` may point at a widget the target app has since + destroyed, and acting on it is a use-after-free in another process's address + space rather than a Python error. + + A context manager because the resolution creates several CoreFoundation + references (the application element, the window, the element itself) and every + one is a Create-Rule +1 that must be released — skipping them measured at + ~400KB of permanent growth per walk. ``snapshot_macos.resolve_element`` owns + that bookkeeping; this wrapper adds the identity check. + + ``None`` is yielded when the index no longer resolves OR when the control now + at that index is not the one the model addressed + (:func:`_same_identity`). The addressing walk uses the widest budgets rather + than the original request's, so the two numberings are not guaranteed + identical when the original walk was truncated — the identity check, not the + index alone, is what makes this safe. It is a second line of defence in any + case: the dispatch chokepoint already refuses on fingerprint drift. + """ + with snapshot_macos.resolve_element(app, rec) as (element, found): + if element is None or found is None or not _same_identity(found, rec): + yield None + return + yield element + + +def _ancestor_press(element: object, rec: ElementRec) -> bool: + """Press the nearest ancestor that plausibly IS the control *rec* names. + + The last rung of the click ladder, and the one that recovers web content. A + clickable row in a WebArea is frequently a plain ``AXStaticText`` inside a + pressable wrapper: the text node advertises no actions and refuses every verb, + so an element click reports failure for a row a human clicks without thinking — + and the model's only remaining move is to guess coordinates, which is what this + whole path exists to avoid. + + **Two guards, because an unguarded version is a wrong-click generator.** + + * *Bounded hops* (:data:`MAX_ANCESTOR_PRESS_HOPS`). Walking to the top would + eventually find something pressable — the page, the window — and press it. + * *Bounded area* (:data:`MAX_ANCESTOR_AREA_RATIO`). The real signal that an + ancestor is "the same control" is that it is roughly the same SIZE. A row + wrapping a text node is a small multiple of its area; a scroll area or page + body is orders of magnitude larger. Without the target's own frame there is + nothing to compare against, so the fallback declines rather than guessing. + + Only ``AXPress`` is attempted, not the full ladder: ``AXOpen`` on a container + can mean something quite different from activating the row inside it + (open the folder vs. select the file), and this rung is already a heuristic — + stacking a second one on top of it widens the blast radius for no clear gain. + + Every ancestor reference is released, including on the success path. + """ + if rec.frame is None: + return False + target_area = _area(rec.frame) + if target_area <= 0: + # A zero-area target makes the ratio test meaningless (everything is + # infinitely larger), so there is no way to bound the walk. Decline. + return False + current = element + owned: list = [] + try: + for _ in range(MAX_ANCESTOR_PRESS_HOPS): + parent = macos_ffi.ax_parent(current) + if parent is None: + return False + owned.append(parent) + current = parent + frame = macos_ffi.ax_frame(parent) + if frame is None: + # No geometry to judge by: keep climbing rather than pressing + # blind. A wrapper with no frame is common in web content and its + # own parent may well have one. + continue + if _area(frame) > target_area * MAX_ANCESTOR_AREA_RATIO: + # Too big to be the control the model addressed, and every further + # ancestor is bigger still — stop, do not keep climbing. + return False + if macos_ffi.ax_perform(parent, AX_PRESS_ACTION) == macos_ffi.AX_ERR_SUCCESS: + return True + return False + finally: + macos_ffi.release_all(owned) + + +def _area(frame: tuple[float, float, float, float]) -> float: + """Area of a ``(x, y, width, height)`` rect, clamped at 0. + + Clamped because AX occasionally reports a negative dimension for an off-screen + or collapsed element, and a negative area would compare as "smaller than + everything" and wave the ancestor guard through. + """ + return max(0.0, frame[2]) * max(0.0, frame[3]) + + +def _same_identity(found: ElementRec, expected: ElementRec) -> bool: + """True when the record at an index is still the control the model addressed. + + Role, subrole and title only — deliberately NOT ``value``. A text field's value + changes as the user types without the control's identity changing at all, and + folding it in would refuse almost every legitimate action. This mirrors + ``render.fingerprint``'s field selection for the same reason, and neither reads + a secure record's bytes. + """ + return ( + found.role == expected.role + and found.subrole == expected.subrole + and found.title == expected.title + ) + + +def _guarded(label: str, run: "Callable[[], DriverResult]") -> DriverResult: + """Run *run*, converting every failure into ``DriverResult(ok=False, ...)``. + + ``ComputerUseError`` carries a message written for a model, so it is passed + through verbatim (without the ``Error: `` prefix — the dispatch layer adds that + exactly once). Anything else is logged with a traceback and reported + generically: an unexpected exception's ``str`` can carry internal paths, and a + model does not benefit from a stack-shaped message. + + A ctypes SEGFAULT is of course NOT catchable here. That is what the + ``argtypes`` discipline and the bounds-guarded array reads in + :mod:`macos_ffi` exist to prevent — this function handles the failures that + *are* representable. + """ + try: + return run() + except ComputerUseError as exc: + return DriverResult(ok=False, text=str(exc)) + except Exception as exc: + logger.warning("computer-use %s failed: %s", label, exc, exc_info=True) + return DriverResult(ok=False, text=f"{label} failed unexpectedly ({type(exc).__name__})") + + +def _pressed_text(index: int, action: str) -> str: + """Result prose that NAMES the winning action. + + The verb is included so a model that needed ``AXOpen`` on a Finder row learns + that from the result instead of rediscovering the ladder on the next element of + the same kind. + """ + verb = "pressed" if action == AX_PRESS_ACTION else f"activated via {action}" + return f"{verb} element {index}" + + +def _ancestor_pressed_text(index: int) -> str: + """Result prose for the ancestor fallback — SAYS that it was the container. + + Not reported as a plain "pressed element N". The model has to know the press + landed on the enclosing control rather than the node it addressed, because that + is the difference between "my click worked" and "something near my click + worked": if the observable outcome is wrong, this sentence is the only clue as + to why, and without it the model would re-try an action that already succeeded + on the wrong target. + """ + return ( + f"element {index} could not be activated directly; pressed its enclosing " + "control instead — verify the result is what you intended" + ) + + +def _sky_click( + app: AppRef, + req: ClickRequest, + left: float, + top: float, + width: float, + height: float, +) -> DriverResult: + """Dispatch one ``sky_click``, converting the screen point to window-local. + + The conversion is the whole reason this helper exists: the window server ignores + the screen coordinate on this path and routes by the WINDOW-LOCAL point written + into the private window-location field (see :mod:`macos_skylight`), so passing + the screen point twice would click the wrong place in the window — off by exactly + the window's origin. + + Both are still sent — the screen point in the event's own position field, the + local one in the private field — because that is the shape the window server + was observed to accept. + """ + assert req.point is not None # the caller checked; kept for the type narrowing + macos_skylight.sky_click( + pid=app.pid, + window_id=app.window_id, + screen_x=req.point[0], + screen_y=req.point[1], + window_x=req.point[0] - left, + window_y=req.point[1] - top, + window_width=width, + window_height=height, + click_count=req.count, + # Passed rather than assumed. The previous call omitted it and the recipe + # built left-button codes unconditionally, so a right-click request through + # this method silently activated the control instead of opening the context + # menu — on a background window the operator cannot see. It is refused (not + # downgraded) inside ``macos_skylight``, and again upstream at the chokepoint + # by ``policy.check_method_button`` so the model gets the legible message. + button=req.button, + ) + return DriverResult(ok=True, text=_click_text(req, app), app=app) + + +def _click_text(req: ClickRequest, app: AppRef) -> str: + """Confirmation prose for a coordinate click, naming the method it used. + + The METHOD is in the text on purpose: a model that asked for ``auto`` needs to + know which path actually ran, because that determines whether a follow-up + should re-aim (an accessibility press landed on a control) or re-measure (a + coordinate click landed on a pixel). It is also the string the operator sees in + a transcript when the agent took their mouse. + """ + point = req.point or (0.0, 0.0) + times = "" if req.count == 1 else f" x{req.count}" + return ( + f"{req.button} click{times} at ({point[0]:.0f}, {point[1]:.0f}) " + f"in '{app.name}' ({req.method})" + ) + + +def _missing_element(app: AppRef, rec: ElementRec) -> DriverResult: + """Refusal for an index that no longer resolves to an element.""" + return DriverResult( + ok=False, + text=_REASON_NO_ELEMENT.format(index=rec.index, app=app.bundle_id or app.name), + ) + + +def _focus_failed(app: AppRef, rec: ElementRec, err: int) -> DriverResult: + """Refusal for keyboard input whose target could not be focused. + + Fail-CLOSED rather than best-effort. ``check_input_target`` validated the element + the model ADDRESSED — ``ElementRec.secure`` is read from that element — so falling + through to app-scoped ``post_text``/``post_key`` after a failed focus delivers the + input to the app's EXISTING focus, which the policy layer never inspected and + which can be the secure field it just refused. Some elements genuinely are not + focusable and did accept typed input before this refusal; that legitimate case is + the cost, and it is recoverable (click the control, then type) whereas a + credential typed into the wrong field is not. + """ + return DriverResult( + ok=False, + text=_REASON_FOCUS_FAILED.format(index=rec.index, app=app.bundle_id or app.name, code=err), + ) + + +def _action_failed(app: AppRef, rec: ElementRec, action: str, err: int) -> DriverResult: + """Refusal for an accessibility action that returned a non-zero code. + + The RAW AX code is included. ``-25205`` in a support thread is immediately + diagnosable ("the element refused an action it advertised"); "the click + failed" is not. + """ + detail = ( + _REASON_UNSUPPORTED_TARGET.format(action=action) + if err == macos_ffi.AX_ERR_ACTION_UNSUPPORTED + else _REASON_AX_ERROR.format(code=err) + ) + return DriverResult( + ok=False, + text=ERR_ACTION_FAILED.format( + action=action, + index=rec.index, + app=app.bundle_id or app.name, + detail=detail, + ), + ) + + +__all__ = ["MacOSBackend"] diff --git a/src/kiro_crew/computer_use/macos_ffi.py b/src/kiro_crew/computer_use/macos_ffi.py new file mode 100644 index 00000000000..724925d1cf2 --- /dev/null +++ b/src/kiro_crew/computer_use/macos_ffi.py @@ -0,0 +1,2040 @@ +"""The ONLY module in this package that touches ctypes. + +Everything native lives here: CoreFoundation memory discipline, the +Accessibility (AX) attribute/action calls, CoreGraphics window enumeration and +event synthesis, and the in-process ImageIO encode. Every other computer-use +module talks to macOS through the helpers below, so the FFI hazards are audited +in one place instead of scattered across five files. + +``import ctypes`` is a top-level import statement (AUTOSDE ``top-level-imports`` +is about statements, not about *loading* a library), but **no ``CDLL`` or +``find_library`` runs at module scope**. The frameworks load inside +:func:`_frameworks`, cached in a module global, and raise +:class:`ComputerUseUnsupported` off macOS — so this module imports cleanly on the +Linux and Windows CI shards and a test can exercise the binding logic with a fake +``CDLL``. + +Six hazards this module exists to contain. Each was reproduced live on a real Mac +(darwin-arm64, Darwin 25.5.0); the failure mode of the first three is a **process +abort**, not a Python exception, so none of them can be handled by a caller. + +1. **A missing ``argtypes`` truncates a 64-bit pointer to 32 bits and + SEGFAULTS.** One omitted ``argtypes`` produced a real ``EXIT=139``. Hence the + declarative :data:`_FN_SPECS` table and a single bind pass: no function in + this module can be reached un-bound, and :func:`_bind` RAISES on + ``argtypes=None`` rather than accepting the ctypes default. +2. **``CFArrayGetValueAtIndex`` past the end raises an UNCATCHABLE ObjC + ``NSRangeException``** — verified ``EXIT=134``, ``libc++abi: terminating``, on + an empty ``CFArray``. ctypes cannot translate an ObjC exception into a Python + one, so every array read goes through :func:`cf_array_items`, which reads the + count FIRST and never indexes out of range. +3. **``CGEventCreateScrollWheelEvent`` is VARIADIC and ctypes marshals its + variadic tail incorrectly on arm64.** Asking for ``(axis1=-3, axis2=0)`` + through the 5-argument form produced ``axis2=30416`` — garbage that would + scroll a live window sideways by an arbitrary amount. Only the fixed + (``wheelCount=1``) prototype is declared, and the second axis is set + afterwards through the non-variadic + ``CGEventSetIntegerValueField`` (verified exact for every input). +4. **``kCFTypeDictionaryKeyCallBacks`` is a STRUCT, not a pointer.** + ``c_void_p.in_dll(...).value`` reads its first word — ``None`` — and passing + that means "no callbacks", i.e. CF neither retains nor releases the dictionary + contents. :func:`_cf_type_dict` passes ``ctypes.addressof`` instead. +5. **A synthesized key event inherits the user's LIVE modifier state.** Typing + ``abc`` produced ``' I Abc'`` until the events were built from a *private* + event source with flags set EXPLICITLY on every event. See :func:`post_key`. +6. **A CFString leak is a real RSS bug.** One tree walk creates thousands of + them, so :func:`cf_string` is a context manager that always ``CFRelease``s. +7. **``CGEventCreateMouseEvent`` takes a ``CGPoint`` BY VALUE**, and there is no + generic "mouse down" event type — the type is PER-BUTTON. Passing two bare + doubles mis-marshals the call under the AArch64 convention, and posting a + right-click as ``kCGEventLeftMouseDown`` with ``button=1`` is delivered as a + LEFT click. Both are handled by :data:`MOUSE_EVENT_TYPES` + + :func:`_mouse_event`. A multi-click is also NOT repeated pairs: it is a pair + carrying ``kCGMouseEventClickState``, which is where AppKit reads + ``NSEvent.clickCount`` from. + +**One deliberate exception to the "``CGEventPostToPid`` only" rule.** +``CGEventPost`` (the global HID tap) and ``CGWarpMouseCursorPosition`` are bound +and are called from :func:`post_mouse_global` / :func:`post_mouse_drag_global` +and nowhere else. Those are the ``click_method: "global"`` path, which +deliberately takes over the operator's physical mouse and is therefore reachable +only when the model NAMED that method — ``auto`` never resolves onto it, and the +resolution happens upstream at the dispatch chokepoint. A test pins the call-site +count, so a stray global post anywhere else in the module fails CI. + +Every AX read is also type-checked with ``CFGetTypeID`` *before* the value is +used: AX attributes are polymorphic and a wrong-type read is another segfault +rather than an exception. +""" + +from __future__ import annotations + +import ctypes +import ctypes.util +import logging +import os +import struct +import tempfile +import threading +import time +from contextlib import contextmanager +from ctypes import ( + POINTER, + c_bool, + c_char_p, + c_double, + c_float, + c_int, + c_int32, + c_int64, + c_long, + c_size_t, + c_ubyte, + c_uint16, + c_uint32, + c_uint64, + c_void_p, +) +from dataclasses import dataclass +from typing import Any, Iterator, Sequence + +from kiro_crew import platform_compat +from kiro_crew.computer_use.types import ( + AX_PARENT, + AX_POSITION, + AX_SIZE, + MOUSE_BUTTON_LEFT, + MOUSE_BUTTON_MIDDLE, + MOUSE_BUTTON_RIGHT, + SCREENSHOT_DIR_NAME, + ComputerUseUnsupported, +) + +logger = logging.getLogger(__name__) + +# ── Library keys (indexes into the bound handle bundle) ── +LIB_CF = "cf" +LIB_AX = "ax" +LIB_CG = "cg" +LIB_IO = "io" +LIB_PROC = "proc" + +# Framework names passed to ``ctypes.util.find_library``. ``proc`` resolves to +# ``/usr/lib/libproc.dylib``; the explicit path is the documented fallback for a +# stripped environment where ``find_library`` returns None. +_FRAMEWORK_NAMES: dict[str, str] = { + LIB_CF: "CoreFoundation", + LIB_AX: "ApplicationServices", + LIB_CG: "CoreGraphics", + LIB_IO: "ImageIO", + LIB_PROC: "proc", +} +_LIBPROC_FALLBACK_PATH = "/usr/lib/libproc.dylib" + +# ── CoreFoundation ABI constants ── +# Stable published values, not derived at runtime. They live here (rather than in +# the platform-free ``types.py``) for the same reason ``keymap.py`` holds the +# CoreGraphics flag masks: they are ABI facts about *this* FFI surface, and a +# reader auditing a call needs the number next to the call. +K_CF_STRING_ENCODING_UTF8 = 0x08000100 +# kCFNumberSInt32Type. Deliberately the sized variant rather than +# kCFNumberIntType(9): both were verified to work for the ImageIO options, and +# the sized name states the C type of the buffer we actually pass. +K_CF_NUMBER_SINT32 = 3 +K_CF_NUMBER_DOUBLE = 13 + +# ``AXValueType`` discriminants (``AXValue.h``). An ``AXValue`` is an opaque box: +# ``AXValueGetValue`` will only unbox it into a buffer of the type it actually +# holds, so the type has to be named at the call. Reading ``AXPosition`` (a +# ``CGPoint``) into a ``CGSize`` buffer would be a silent field transposition — +# ``y`` read as ``width`` — so the discriminant is passed explicitly and the call +# is checked rather than assumed. +K_AX_VALUE_CG_POINT = 1 +K_AX_VALUE_CG_SIZE = 2 + +# ── Accessibility error codes ── +AX_ERR_SUCCESS = 0 +# kAXErrorCannotComplete. Electron/Chromium apps answer this to EVERY attribute +# read until ``AXManualAccessibility`` is set. Not a fatal error — a retry signal. +AX_ERR_CANNOT_COMPLETE = -25204 +# kAXErrorActionUnsupported. Returned by an element that ADVERTISED the action: +# a real ``AXScrollArea`` listed ``AXScrollDownByPage`` and then refused it. The +# advertised action list is a hint, never a contract. +# +# The two codes below are DISTINCT and were both confirmed empirically on this +# macOS rather than copied from a header (they had been transposed onto the same +# value, which made the scroll fallback's "was it the action that was refused?" +# test answer for the wrong error): +# AXUIElementPerformAction(app, "AXNotARealAction") -> -25206 +# AXUIElementCopyAttributeValue(app, "AXNotAnAttribute") -> -25212 +AX_ERR_ACTION_UNSUPPORTED = -25206 +# kAXErrorAttributeUnsupported. Extremely common and completely benign — window +# elements answer it for ``AXValue``/``AXEnabled``, and every application element +# on this macOS answers it for ``AXBundleIdentifier`` (see ``apps_macos``). +AX_ERR_ATTRIBUTE_UNSUPPORTED = -25205 +AX_ERR_NO_VALUE = -25212 + +# The Electron/Chromium accessibility opt-in, plus its bounded wait. Measured: +# Obsidian exposed a 13-node stub before the opt-in and 574 nodes after it; +# Chrome went from a hard ``-25204`` to 1,431 nodes. +AX_MANUAL_ACCESSIBILITY = "AXManualAccessibility" +ELECTRON_OPT_IN_WAIT_SECS = 2.0 +ELECTRON_OPT_IN_POLL_SECS = 0.25 +# MANDATORY, immediately after creating the application element. ctypes releases +# the GIL around the C call, so an unresponsive target app would otherwise park +# the calling worker thread indefinitely with no way to interrupt it. +AX_MESSAGING_TIMEOUT_SECS = 2.0 + +# ── CoreGraphics window-list ABI constants ── +K_CG_WINDOW_LIST_ON_SCREEN_ONLY = 1 +K_CG_WINDOW_LIST_EXCLUDE_DESKTOP = 16 +K_CG_WINDOW_LIST_INCLUDING_WINDOW = 8 +K_CG_WINDOW_IMAGE_BOUNDS_IGNORE_FRAMING = 1 +K_CG_NULL_WINDOW_ID = 0 + +# Window-list dictionary keys. Verified present on this macOS: the dict carries +# exactly {Alpha, Bounds, IsOnscreen, Layer, MemoryUsage, Name, Number, +# OwnerName, OwnerPID, SharingState, StoreType} — note there is NO bundle-id key, +# which is why ``apps_macos`` resolves the bundle id from the executable path. +CG_WINDOW_NUMBER = "kCGWindowNumber" +CG_WINDOW_OWNER_PID = "kCGWindowOwnerPID" +CG_WINDOW_OWNER_NAME = "kCGWindowOwnerName" +CG_WINDOW_NAME = "kCGWindowName" +CG_WINDOW_LAYER = "kCGWindowLayer" +# The window rect, as a CFDictionary of {X, Y, Width, Height} in TOP-LEFT screen +# coordinates (CoreGraphics' own convention for this key). +CG_WINDOW_BOUNDS = "kCGWindowBounds" +CG_BOUNDS_X = "X" +CG_BOUNDS_Y = "Y" +CG_BOUNDS_W = "Width" +CG_BOUNDS_H = "Height" +# Only layer 0 is a real application window; menu bars, the Dock, tooltips and +# shadows live on other layers and their owning pid is often a system agent. +CG_WINDOW_LAYER_NORMAL = 0 + +# ── CoreGraphics event ABI constants ── +# kCGEventSourceStatePrivate. NEVER pass NULL: an event built from the default +# (HID) source inherits the user's live modifier state. +K_CG_EVENT_SOURCE_STATE_PRIVATE = 1 +# kCGScrollEventUnitLine. The enum has exactly TWO members — ``Pixel`` (0) and +# ``Line`` (1); there is no page unit, so a "page" of scrolling is expressed as a +# line count in the delta, not by the unit. Named accurately here because the old +# spelling (``…_UNIT_PAGE``) read as though the OS were doing the page conversion +# and invited a future caller to pass a page COUNT as the delta. +K_CG_SCROLL_EVENT_UNIT_LINE = 1 +#: Lines of scrolling treated as one "page" for the wheel-event fallback. The AX +#: page actions are tried first (``macos_driver.scroll``); this is only the +#: synthesized approximation for elements that refuse them. +CG_SCROLL_LINES_PER_PAGE = 10 +# kCGScrollWheelEventDeltaAxis1 (vertical) / Axis2 (horizontal). Set through +# ``CGEventSetIntegerValueField`` because the multi-axis constructor is variadic. +K_CG_SCROLL_DELTA_AXIS_1 = 11 +K_CG_SCROLL_DELTA_AXIS_2 = 12 + +# ── CoreGraphics mouse-event ABI constants ── +# ``CGEventType`` members. There is NO generic "mouse down" — the type is +# PER-BUTTON, and a right-click posted as ``kCGEventLeftMouseDown`` with +# ``button=1`` is silently delivered as a LEFT click by AppKit. Hence the explicit +# per-button triples in :data:`MOUSE_EVENT_TYPES` rather than one type plus a +# button number. +K_CG_EVENT_LEFT_MOUSE_DOWN = 1 +K_CG_EVENT_LEFT_MOUSE_UP = 2 +K_CG_EVENT_RIGHT_MOUSE_DOWN = 3 +K_CG_EVENT_RIGHT_MOUSE_UP = 4 +K_CG_EVENT_MOUSE_MOVED = 5 +K_CG_EVENT_LEFT_MOUSE_DRAGGED = 6 +K_CG_EVENT_RIGHT_MOUSE_DRAGGED = 7 +K_CG_EVENT_OTHER_MOUSE_DOWN = 25 +K_CG_EVENT_OTHER_MOUSE_UP = 26 +K_CG_EVENT_OTHER_MOUSE_DRAGGED = 27 + +# ``CGMouseButton`` members, paired with the event types above. +K_CG_MOUSE_BUTTON_LEFT = 0 +K_CG_MOUSE_BUTTON_RIGHT = 1 +K_CG_MOUSE_BUTTON_CENTER = 2 + +#: ``button_name -> (button_number, down_type, up_type, dragged_type)``. +#: Keyed by the SAME strings ``types.MOUSE_BUTTONS`` publishes, so the wire +#: vocabulary and the ABI mapping meet in exactly one place. Middle-click uses the +#: ``OtherMouse`` family, which is what "center" means in ``CGMouseButton``. +MOUSE_EVENT_TYPES: dict[str, tuple[int, int, int, int]] = { + MOUSE_BUTTON_LEFT: ( + K_CG_MOUSE_BUTTON_LEFT, + K_CG_EVENT_LEFT_MOUSE_DOWN, + K_CG_EVENT_LEFT_MOUSE_UP, + K_CG_EVENT_LEFT_MOUSE_DRAGGED, + ), + MOUSE_BUTTON_RIGHT: ( + K_CG_MOUSE_BUTTON_RIGHT, + K_CG_EVENT_RIGHT_MOUSE_DOWN, + K_CG_EVENT_RIGHT_MOUSE_UP, + K_CG_EVENT_RIGHT_MOUSE_DRAGGED, + ), + MOUSE_BUTTON_MIDDLE: ( + K_CG_MOUSE_BUTTON_CENTER, + K_CG_EVENT_OTHER_MOUSE_DOWN, + K_CG_EVENT_OTHER_MOUSE_UP, + K_CG_EVENT_OTHER_MOUSE_DRAGGED, + ), +} + +# kCGMouseEventClickState. A double click is NOT two down/up pairs — it is a pair +# whose click state is 2, and AppKit ignores the repetition otherwise (verified +# behaviour of every Cocoa control: ``NSEvent.clickCount`` is read from this +# field). So the count is written onto each event rather than expressed by +# posting more of them. +K_CG_MOUSE_EVENT_CLICK_STATE = 1 + +# kCGHIDEventTap — the destination ``CGEventPost`` delivers to. Bound (and used) +# ONLY on the ``global`` click path, which is the one path that deliberately +# targets the system-wide tap because it is simulating a physical mouse; every +# other path uses ``CGEventPostToPid``. See :func:`post_mouse_global`. +K_CG_HID_EVENT_TAP = 0 + +#: Intermediate ``MouseDragged`` events synthesized between a drag's endpoints. +#: A bare down-at-A / up-at-B pair is NOT a drag to most apps: a canvas records +#: two isolated points and a text view selects nothing, because the gesture is +#: recognized from the intermediate motion. 6 steps is what the live prototype +#: needed for TextEdit to register a selection. +DRAG_STEPS = 6 +#: Pause between synthesized drag events. Without it every event carries an +#: identical timestamp and AppKit's gesture recognizer coalesces them into one +#: motion, which is the same silent no-op as sending no steps at all. +DRAG_STEP_DELAY_SECS = 0.02 +#: Pause between the two halves of a click pair, and between repeated clicks of a +#: multi-click. Below the OS double-click interval so a ``click_count=2`` is +#: recognized as a double click rather than as two separate clicks. +CLICK_PAIR_DELAY_SECS = 0.01 + +# ── ImageIO ── +UTI_JPEG = "public.jpeg" +IO_KEY_MAX_PIXEL_SIZE = "kCGImageDestinationImageMaxPixelSize" +IO_KEY_LOSSY_QUALITY = "kCGImageDestinationLossyCompressionQuality" + +# ``proc_pidpath`` buffer size (PROC_PIDPATHINFO_MAXSIZE = 4 * MAXPATHLEN). +PROC_PIDPATH_MAX = 4096 + +# Sentinel for "this symbol returns void". ``restype = None`` is *meaningful* to +# ctypes (it means "no return value"), so it cannot double as "unspecified" — +# without a distinct sentinel a typo'd table row would silently bind a void +# restype onto a pointer-returning function and truncate the result. +_VOID = object() + +# ``CFTypeID`` is a ``unsigned long`` (64-bit on every supported Mac). Aliased so +# the table below states the CF type name rather than repeating a width, and so a +# future 32-bit target is one edit. +_CF_TYPE_ID = c_uint64 + + +class CGPoint(ctypes.Structure): + """CoreGraphics ``CGPoint``. Declared as a struct, never as two doubles.""" + + _fields_ = [("x", c_double), ("y", c_double)] + + +class CGSize(ctypes.Structure): + """CoreGraphics ``CGSize``.""" + + _fields_ = [("width", c_double), ("height", c_double)] + + +class CGRect(ctypes.Structure): + """CoreGraphics ``CGRect``. + + Passing four bare doubles where a nested struct is expected mis-marshals the + call on both arm64 and x86_64 — the register/stack layout of a struct + argument is not the layout of its flattened members. + """ + + _fields_ = [("origin", CGPoint), ("size", CGSize)] + + +# ``(lib_key, symbol, restype, argtypes)``. ``argtypes`` is MANDATORY on every +# row; :func:`_bind` raises when it is None. Bound in one pass at first +# :func:`_frameworks` call so no symbol can be reached un-bound. +# +# ``CGEventPostToPid`` is the default and the ONLY delivery for every keyboard, +# scroll and app-scoped mouse path: it targets the app the model addressed rather +# than whatever window the user is actually working in. ``CGEventPost`` (the +# global HID tap) is ALSO bound, but reached from exactly one function — +# :func:`post_mouse_global` — because the ``global`` click method deliberately +# simulates a physical mouse and there is no per-pid form of that. A test pins +# that call-site count at one, so a stray ``CGEventPost`` anywhere else fails CI. +# ``CGWarpMouseCursorPosition`` is in the same category: bound, and called from +# that one function only. +_FN_SPECS: tuple[tuple[str, str, Any, list[Any]], ...] = ( + # ── CoreFoundation: memory + type identity ── + (LIB_CF, "CFRelease", _VOID, [c_void_p]), + (LIB_CF, "CFRetain", c_void_p, [c_void_p]), + (LIB_CF, "CFGetTypeID", _CF_TYPE_ID, [c_void_p]), + # Identity, not pointer equality: two DIFFERENT ``AXUIElement`` refs can name + # the same UI element (``AXFocusedUIElement`` and the same node reached by + # walking ``AXChildren`` are distinct allocations), and comparing the raw + # addresses would answer "not the same" for the element the caret is in. + (LIB_CF, "CFEqual", c_bool, [c_void_p, c_void_p]), + # ── CoreFoundation: strings ── + (LIB_CF, "CFStringCreateWithCString", c_void_p, [c_void_p, c_char_p, c_uint32]), + # CFIndex is a signed long (64-bit here), NOT an int32 — a 32-bit restype + # would truncate the length of a large AXValue and undersize the buffer. + (LIB_CF, "CFStringGetLength", c_long, [c_void_p]), + (LIB_CF, "CFStringGetCString", c_bool, [c_void_p, c_char_p, c_long, c_uint32]), + (LIB_CF, "CFStringGetTypeID", _CF_TYPE_ID, []), + # ── CoreFoundation: arrays ── + (LIB_CF, "CFArrayGetTypeID", _CF_TYPE_ID, []), + (LIB_CF, "CFArrayGetCount", c_long, [c_void_p]), + (LIB_CF, "CFArrayGetValueAtIndex", c_void_p, [c_void_p, c_long]), + # ── CoreFoundation: dictionaries ── + (LIB_CF, "CFDictionaryGetTypeID", _CF_TYPE_ID, []), + (LIB_CF, "CFDictionaryGetValue", c_void_p, [c_void_p, c_void_p]), + (LIB_CF, "CFDictionaryCreateMutable", c_void_p, [c_void_p, c_long, c_void_p, c_void_p]), + (LIB_CF, "CFDictionarySetValue", _VOID, [c_void_p, c_void_p, c_void_p]), + # ── CoreFoundation: numbers + booleans ── + (LIB_CF, "CFNumberGetTypeID", _CF_TYPE_ID, []), + (LIB_CF, "CFNumberCreate", c_void_p, [c_void_p, c_int, c_void_p]), + (LIB_CF, "CFNumberGetValue", c_bool, [c_void_p, c_int, c_void_p]), + (LIB_CF, "CFBooleanGetTypeID", _CF_TYPE_ID, []), + (LIB_CF, "CFBooleanGetValue", c_bool, [c_void_p]), + # ── CoreFoundation: data (the ImageIO encode sink) ── + (LIB_CF, "CFDataCreateMutable", c_void_p, [c_void_p, c_long]), + (LIB_CF, "CFDataGetLength", c_long, [c_void_p]), + (LIB_CF, "CFDataGetBytePtr", POINTER(c_ubyte), [c_void_p]), + # ── ApplicationServices (Accessibility) ── + (LIB_AX, "AXIsProcessTrusted", c_bool, []), + (LIB_AX, "AXUIElementGetTypeID", _CF_TYPE_ID, []), + (LIB_AX, "AXUIElementCreateApplication", c_void_p, [c_int32]), + (LIB_AX, "AXUIElementCopyAttributeValue", c_int32, [c_void_p, c_void_p, POINTER(c_void_p)]), + (LIB_AX, "AXUIElementSetAttributeValue", c_int32, [c_void_p, c_void_p, c_void_p]), + (LIB_AX, "AXUIElementCopyActionNames", c_int32, [c_void_p, POINTER(c_void_p)]), + (LIB_AX, "AXUIElementPerformAction", c_int32, [c_void_p, c_void_p]), + # ``AXValue`` unboxing, for ``AXPosition``/``AXSize``. ``AXValueGetType`` + # is checked BEFORE ``AXValueGetValue`` so a mistyped box is refused rather + # than transposed into the wrong struct fields. + (LIB_AX, "AXValueGetTypeID", _CF_TYPE_ID, []), + (LIB_AX, "AXValueGetType", c_int32, [c_void_p]), + (LIB_AX, "AXValueGetValue", c_bool, [c_void_p, c_int32, c_void_p]), + # Settability, the only way to tell a read-only text field from an editable + # one — an ``AXTextField`` the app has disabled for input still reports + # ``AXEnabled=true`` and still advertises a value. + (LIB_AX, "AXUIElementIsAttributeSettable", c_int32, [c_void_p, c_void_p, POINTER(c_bool)]), + # Takes a C ``float``, not a double — a c_double here passes the value in the + # wrong register class and the timeout is never applied. + (LIB_AX, "AXUIElementSetMessagingTimeout", c_int32, [c_void_p, c_float]), + # ── CoreGraphics: window list + capture ── + (LIB_CG, "CGWindowListCopyWindowInfo", c_void_p, [c_uint32, c_uint32]), + (LIB_CG, "CGWindowListCreateImage", c_void_p, [CGRect, c_uint32, c_uint32, c_uint32]), + (LIB_CG, "CGImageGetWidth", c_size_t, [c_void_p]), + (LIB_CG, "CGImageGetHeight", c_size_t, [c_void_p]), + (LIB_CG, "CGImageRelease", _VOID, [c_void_p]), + # Preflight ONLY. ``CGRequestScreenCaptureAccess`` pops a system dialog from + # whatever process asks, which from a background sidecar is an unexplained + # prompt the operator cannot attribute — never called anywhere. + (LIB_CG, "CGPreflightScreenCaptureAccess", c_bool, []), + # ── CoreGraphics: event synthesis ── + (LIB_CG, "CGEventSourceCreate", c_void_p, [c_int32]), + (LIB_CG, "CGEventCreateKeyboardEvent", c_void_p, [c_void_p, c_uint16, c_bool]), + (LIB_CG, "CGEventKeyboardSetUnicodeString", _VOID, [c_void_p, c_long, POINTER(c_uint16)]), + # The FIXED-arity prototype only: ``wheelCount`` is pinned to 1 by every + # caller and the second axis is applied through CGEventSetIntegerValueField. + # Declaring the variadic tail produced garbage on arm64 (hazard 3 above). + (LIB_CG, "CGEventCreateScrollWheelEvent", c_void_p, [c_void_p, c_uint32, c_uint32, c_int32]), + (LIB_CG, "CGEventSetIntegerValueField", _VOID, [c_void_p, c_uint32, c_int64]), + (LIB_CG, "CGEventSetFlags", _VOID, [c_void_p, c_uint64]), + (LIB_CG, "CGEventPostToPid", _VOID, [c_int32, c_void_p]), + # ── CoreGraphics: mouse-event synthesis ── + # ``CGPoint`` BY VALUE. Declaring the location as two bare doubles + # mis-marshals the whole call under the AArch64 calling convention — the + # button argument lands in the wrong register and the event is created for an + # arbitrary button at an arbitrary point. Verified in the live prototype: the + # struct argtype is mandatory, not documentation. + (LIB_CG, "CGEventCreateMouseEvent", c_void_p, [c_void_p, c_uint32, CGPoint, c_uint32]), + # The GLOBAL HID tap. Reached ONLY from ``post_mouse_global`` — the + # ``click_method: "global"`` path, which the model must name explicitly. There + # is no per-pid form of "simulate a physical mouse", which is exactly why that + # method has to be asked for by name and every other path does not. + (LIB_CG, "CGEventPost", _VOID, [c_uint32, c_void_p]), + # Warps the operator's PHYSICAL cursor. Same one call site, same rule. + # ``CGPoint`` by value again. + (LIB_CG, "CGWarpMouseCursorPosition", c_int32, [CGPoint]), + # ── ImageIO ── + ( + LIB_IO, + "CGImageDestinationCreateWithData", + c_void_p, + [c_void_p, c_void_p, c_size_t, c_void_p], + ), + (LIB_IO, "CGImageDestinationAddImage", _VOID, [c_void_p, c_void_p, c_void_p]), + (LIB_IO, "CGImageDestinationFinalize", c_bool, [c_void_p]), + # ── libproc ── + (LIB_PROC, "proc_pidpath", c_int, [c_int, c_void_p, c_uint32]), +) + +# Symbols that are bound if present and simply absent otherwise. Same +# ``argtypes``-mandatory discipline; the only difference is that a missing symbol +# is not a failure. +# +# ``_AXUIElementGetWindow`` is the only way to learn which CoreGraphics window an +# ``AXUIElement`` corresponds to, and correlating the two is a correctness +# requirement, not a nicety: ``AXWindows[0]`` and the CoreGraphics window list are +# INDEPENDENTLY ordered. Verified live on Notes — ``AXWindows[0]`` was an +# ``AXDialog`` with an empty title (3 nodes) while the frontmost CG window was the +# real 103-node document window, so walking ``AXWindows[0]`` and capturing the CG +# frontmost window would have returned a tree and an image of DIFFERENT WINDOWS. +# The leading underscore marks it private-but-stable (it has shipped for a decade +# and every accessibility client uses it); it is optional here so a future macOS +# that removes it degrades to the title/AXMain fallback in ``snapshot_macos`` +# rather than breaking the feature. +_OPTIONAL_FN_SPECS: tuple[tuple[str, str, Any, list[Any]], ...] = ( + (LIB_AX, "_AXUIElementGetWindow", c_int32, [c_void_p, POINTER(c_uint32)]), +) + + +@dataclass(frozen=True) +class Libs: + """The five bound framework handles. + + Frozen and built exactly once: rebinding on every call would re-run 50 + ``getattr`` + attribute assignments per tree walk, and a partially bound + handle is the hazard the single bind pass exists to prevent. + """ + + cf: Any + ax: Any + cg: Any + io: Any + proc: Any + + +@dataclass(frozen=True) +class TypeIds: + """Cached ``CFTypeID`` values for the types we accept from AX and CF. + + Cached because every attribute read compares against them; each is a C call + and a tree walk performs thousands of reads. + """ + + string: int + array: int + dictionary: int + number: int + boolean: int + ax_element: int + # ``AXValue`` — the opaque box ``AXPosition``/``AXSize`` come wrapped in. A + # separate id from ``ax_element``: handing an ``AXValue`` to an + # element-expecting call (or the reverse) is a segfault, not a type error. + ax_value: int + + +@dataclass(frozen=True) +class WindowInfo: + """One entry of the CoreGraphics on-screen window list.""" + + window_id: int + pid: int + owner_name: str + title: str + layer: int + # Screen rect in TOP-LEFT coordinates, or ``None`` when the entry carried no + # usable ``kCGWindowBounds``. Needed to answer "which app owns this pixel?", + # which is what confines a real-pointer click to the authorized application. + bounds: "tuple[float, float, float, float] | None" = None + + +_libs: "Libs | None" = None +_type_ids: "TypeIds | None" = None +_event_source: "c_void_p | None" = None +# One lock for the lazy singletons. The MCP dispatch layer is single-threaded +# today, but the gateway offloads snapshots to an executor, so two threads can +# reach first-use concurrently and a double bind would leak an event source. +_init_lock = threading.Lock() + + +def _bind(lib: Any, symbol: str, restype: Any, argtypes: "Sequence[Any] | None") -> None: + """Set BOTH ``restype`` and ``argtypes`` on one symbol, or raise. + + ``argtypes is None`` is a programming error, not a permissive default: a + Python int passed to an un-declared parameter is marshalled as a 32-bit C + ``int``, which TRUNCATES a 64-bit pointer and segfaults the process. That is + not a theoretical concern — it produced a real ``EXIT=139`` during + prototyping, and a segfault inside ctypes cannot be caught, logged or + retried. + + ``restype`` uses the :data:`_VOID` sentinel for void-returning functions so + "returns nothing" is distinguishable from "row is incomplete". + """ + if argtypes is None: + raise ComputerUseUnsupported( + f"computer-use FFI table is invalid: {symbol} has no argtypes. Every " + "bound symbol MUST declare argtypes — an undeclared pointer argument " + "is marshalled as a 32-bit int and segfaults the process." + ) + fn = getattr(lib, symbol) + fn.restype = None if restype is _VOID else restype + fn.argtypes = list(argtypes) + + +def _load_library(key: str) -> Any: + """Load one framework by key, raising :class:`ComputerUseUnsupported`. + + ``find_library`` is called here — inside a function — and never at module + scope: at module scope it would run on the Linux CI fleet at import time and + break collection of every test that transitively imports this package. + """ + name = _FRAMEWORK_NAMES[key] + path = ctypes.util.find_library(name) + if path is None and key == LIB_PROC: + # libproc is always present on macOS but ``find_library`` can miss it in + # a stripped environment; the absolute path is the documented location. + path = _LIBPROC_FALLBACK_PATH + if path is None: + raise ComputerUseUnsupported(f"macOS framework {name} could not be located") + try: + return ctypes.CDLL(path) + except OSError as exc: + raise ComputerUseUnsupported(f"macOS framework {name} could not be loaded: {exc}") from exc + + +def _frameworks() -> Libs: + """Load + bind the five frameworks once, then return the cached bundle. + + NOT at module scope. A module-level ``CDLL`` would raise ``OSError`` on a + non-macOS runner during import and take down test collection for the whole + package; here the failure is a typed :class:`ComputerUseUnsupported` that the + backend seam converts into a refusal. + """ + global _libs, _type_ids + if _libs is not None: + return _libs + with _init_lock: + if _libs is not None: # pragma: no cover - double-checked under the lock + return _libs + if not platform_compat.IS_MACOS: + raise ComputerUseUnsupported( + "the macOS computer-use driver requires macOS (ApplicationServices " + "and CoreGraphics do not exist on this platform)" + ) + handles = {key: _load_library(key) for key in _FRAMEWORK_NAMES} + # ONE bind pass over the whole table before anything is callable, so a + # missing symbol or a missing argtypes row fails here rather than at the + # first call from deep inside a tree walk. + for lib_key, symbol, restype, argtypes in _FN_SPECS: + _bind(handles[lib_key], symbol, restype, argtypes) + for lib_key, symbol, restype, argtypes in _OPTIONAL_FN_SPECS: + try: + _bind(handles[lib_key], symbol, restype, argtypes) + except AttributeError: + # Absent on this macOS: the caller has a documented fallback. + logger.debug("optional computer-use symbol %s is unavailable", symbol) + libs = Libs( + cf=handles[LIB_CF], + ax=handles[LIB_AX], + cg=handles[LIB_CG], + io=handles[LIB_IO], + proc=handles[LIB_PROC], + ) + _type_ids = TypeIds( + string=int(libs.cf.CFStringGetTypeID()), + array=int(libs.cf.CFArrayGetTypeID()), + dictionary=int(libs.cf.CFDictionaryGetTypeID()), + number=int(libs.cf.CFNumberGetTypeID()), + boolean=int(libs.cf.CFBooleanGetTypeID()), + ax_element=int(libs.ax.AXUIElementGetTypeID()), + ax_value=int(libs.ax.AXValueGetTypeID()), + ) + _libs = libs + return libs + + +def frameworks() -> Libs: + """Public accessor for the bound framework bundle (see :func:`_frameworks`).""" + return _frameworks() + + +def type_ids() -> TypeIds: + """Cached ``CFTypeID`` values; loads the frameworks on first use.""" + _frameworks() + assert _type_ids is not None # established by _frameworks() + return _type_ids + + +def reset_frameworks() -> None: + """Drop the cached handles, type ids and event source. + + For the swap seam and for tests that install a fake ``CDLL``: without this a + test's fake would never be bound because the real handles are already cached. + Releases the event source rather than leaking it. + """ + global _libs, _type_ids, _event_source + with _init_lock: + if _event_source is not None and _libs is not None: + try: + _libs.cf.CFRelease(_event_source) + except Exception: + logger.debug("event source release failed", exc_info=True) + _event_source = None + _libs = None + _type_ids = None + + +def available() -> bool: + """True when the frameworks load here. Never raises.""" + try: + _frameworks() + return True + except Exception: + return False + + +# ── CoreFoundation helpers ── + + +@contextmanager +def cf_string(text: str) -> Iterator[c_void_p]: + """Create a CFString for *text* and ALWAYS ``CFRelease`` it on exit. + + A context manager rather than a plain factory because a single tree walk + creates thousands of these (one per attribute name per node): leaking them is + a genuine RSS bug the session watchdog would eventually recycle the process + over, and a bare ``try/finally`` at every call site would be forgotten once. + + Yields a NULL ``c_void_p`` when creation fails, which every caller treats as + "attribute unavailable" — CF calls tolerate NULL, so a failed allocation + degrades to a missing value rather than a crash. + """ + libs = _frameworks() + ref = c_void_p( + libs.cf.CFStringCreateWithCString(None, text.encode("utf-8"), K_CF_STRING_ENCODING_UTF8) + ) + try: + yield ref + finally: + if ref: + libs.cf.CFRelease(ref) + + +def cf_string_value(ref: Any) -> str: + """Decode a CFString into ``str``, or ``""``. + + Sizes the buffer from ``CFStringGetLength`` * 4 + 1: the length is in UTF-16 + code units and a code unit can expand to at most 4 UTF-8 bytes (surrogate + pairs count as two units, so this is a safe over-estimate). Undersizing makes + ``CFStringGetCString`` fail, which would silently drop the value. + + **The type check is a crash guard, not a nicety, and it lives HERE rather than + at each call site.** ``CFStringGetLength`` on a non-CFString raises an ObjC + ``NSInvalidArgumentException`` (verified on darwin-arm64: + ``-[__NSCFNumber length]: unrecognized selector`` -> ``libc++abi: terminating``, + exit 134). ctypes cannot translate an ObjC exception into a Python one, so it + is an UNCATCHABLE process abort — ``macos_driver._guarded`` cannot stop it and + the whole gateway dies with every session on it. The contents of an AX array are + arbitrary app-controlled data (``AXActionNames`` really does carry odd payloads; + see ``_sanitize_action``), so a caller that forgets the check is a + remote-crash primitive. Centralising it means no future reader can reintroduce + the hazard; the redundant ``cf_is`` checks at the typed readers stay as cheap + short-circuits. + """ + if not ref: + return "" + libs = _frameworks() + if not cf_is(ref, type_ids().string): + return "" + size = int(libs.cf.CFStringGetLength(ref)) * 4 + 1 + if size <= 1: + return "" + buf = ctypes.create_string_buffer(size) + if not libs.cf.CFStringGetCString(ref, buf, size, K_CF_STRING_ENCODING_UTF8): + return "" + return buf.value.decode("utf-8", "replace") + + +def cf_is(ref: Any, type_id: int) -> bool: + """True when *ref* is non-NULL and its ``CFTypeID`` equals *type_id*. + + Called before EVERY use of an AX-returned value. AX attributes are + polymorphic (``AXValue`` can be a string, a number, an ``AXValue`` struct or + an element) and handing a CFNumber to ``CFStringGetCString`` is a segfault, + not a ``TypeError``. + """ + if not ref: + return False + return int(_frameworks().cf.CFGetTypeID(ref)) == type_id + + +def cf_array_items(ref: Any, *, limit: int = 0) -> list[Any]: + """Return a CFArray's elements as a list, or ``[]`` for a non-array. + + **The count is read first and never exceeded.** Indexing a CFArray past its + end raises an ObjC ``NSRangeException``, which ctypes cannot translate: the + process aborts with ``libc++abi: terminating due to uncaught exception`` + (reproduced live, ``EXIT=134``, on an empty array). Since a caller cannot + recover from that, no caller is allowed to index an array directly. + + *limit* caps how many entries are materialised, so a pathological child list + cannot blow memory before the walk's own node budget notices. + """ + if not cf_is(ref, type_ids().array): + return [] + libs = _frameworks() + count = int(libs.cf.CFArrayGetCount(ref)) + if count <= 0: + return [] + if limit > 0: + count = min(count, limit) + return [libs.cf.CFArrayGetValueAtIndex(ref, i) for i in range(count)] + + +def cf_number_int(ref: Any) -> "int | None": + """Read a CFNumber as a 32-bit int, or ``None`` for a non-number.""" + if not cf_is(ref, type_ids().number): + return None + out = c_int32() + if not _frameworks().cf.CFNumberGetValue(ref, K_CF_NUMBER_SINT32, ctypes.byref(out)): + return None + return int(out.value) + + +def cf_number_double(ref: Any) -> "float | None": + """Read a CFNumber as a double, or ``None`` for a non-number. + + Separate from :func:`cf_number_int` because the window-list ``kCGWindowBounds`` + components are CGFloats: reading them through the SInt32 path would truncate a + fractional origin (Retina windows routinely sit on half-pixel boundaries) and + could place a rect edge a pixel away from where it really is — which matters + when the rect is being used to decide whether a click is inside an authorized + application's window. + """ + if not cf_is(ref, type_ids().number): + return None + out = c_double() + if not _frameworks().cf.CFNumberGetValue(ref, K_CF_NUMBER_DOUBLE, ctypes.byref(out)): + return None + return float(out.value) + + +def cf_bool_value(ref: Any) -> "bool | None": + """Read a CFBoolean, or ``None`` for a non-boolean.""" + if not cf_is(ref, type_ids().boolean): + return None + return bool(_frameworks().cf.CFBooleanGetValue(ref)) + + +def cf_true() -> c_void_p: + """``kCFBooleanTrue``. + + Read through ``c_void_p.in_dll``: the symbol IS a pointer to the shared + singleton, so ``.value`` is the right read here — unlike the dictionary + callback STRUCTS, which need :func:`_cf_type_dict`'s ``addressof``. + """ + return c_void_p.in_dll(_frameworks().cf, "kCFBooleanTrue") + + +def _cf_type_dict() -> c_void_p: + """Create an empty mutable CFDictionary with the standard CFType callbacks. + + ``kCFTypeDictionaryKeyCallBacks`` is a ``CFDictionaryKeyCallBacks`` STRUCT, + not a pointer to one. ``c_void_p.in_dll(...).value`` therefore reads the + struct's first word — verified ``None`` — and passing that means "no + callbacks": CF would neither retain the keys/values we insert nor release + them, so a CFNumber we release becomes a dangling entry. ``addressof`` on the + ``in_dll`` view yields the address of the framework's own static storage, + which is what the API actually wants and stays valid for the process + lifetime. + """ + libs = _frameworks() + key_cb = c_void_p.in_dll(libs.cf, "kCFTypeDictionaryKeyCallBacks") + value_cb = c_void_p.in_dll(libs.cf, "kCFTypeDictionaryValueCallBacks") + return c_void_p( + libs.cf.CFDictionaryCreateMutable( + None, 0, ctypes.addressof(key_cb), ctypes.addressof(value_cb) + ) + ) + + +# ── Accessibility helpers ── + + +def ax_app_element(pid: int) -> c_void_p: + """Create the application-level ``AXUIElement`` for *pid*. **CALLER OWNS it.** + + ``AXUIElementCreateApplication`` is a Create-Rule call (+1, verified retain + count 1), so the caller must ``CFRelease``. Prefer :func:`ax_application`, + which does it. + + The messaging timeout is set here — immediately after creation and before any + attribute read — because that is the ONLY place it can be guaranteed. Every + subsequent AX call inherits it, and without it a hung target app parks the + calling thread with no interruption path (ctypes releases the GIL for the + duration of the C call, so not even a signal handler runs). + """ + libs = _frameworks() + elem = c_void_p(libs.ax.AXUIElementCreateApplication(int(pid))) + if elem: + libs.ax.AXUIElementSetMessagingTimeout(elem, c_float(AX_MESSAGING_TIMEOUT_SECS)) + return elem + + +@contextmanager +def ax_application(pid: int) -> Iterator[c_void_p]: + """:func:`ax_app_element` with the mandatory release. + + Measured leak without it: 1.4MB per 30,000 creations. One per snapshot is not + dramatic on its own, but a long-lived sidecar taking a snapshot every turn + accumulates it forever, and the context-manager form makes the obligation + impossible to forget. + """ + libs = _frameworks() + elem = ax_app_element(pid) + try: + yield elem + finally: + if elem: + libs.cf.CFRelease(elem) + + +def ax_attr(elem: Any, name: str) -> "tuple[Any, int]": + """Read one AX attribute: ``(value_or_None, ax_error)``. **CALLER OWNS the value.** + + ``AXUIElementCopyAttributeValue`` follows CoreFoundation's **Create Rule**: the + name contains "Copy", so the returned reference is +1 and the caller MUST + ``CFRelease`` it. Verified: the returned CFString had a retain count of 1, and + 30,000 reads without a release grew RSS by 9.4MB while the same 30,000 with a + release grew it by 0.0MB. + + This is a genuine leak, not a theoretical one — a single 1,200-node walk + performs ~5,000 of these reads, which measured at ~400KB of permanent growth + per walk before the fix. A long session would have been recycled by the + watchdog on RSS. + + Prefer :func:`ax_owned_attr` (a context manager that releases) or the typed + helpers built on it. This raw form exists for the two callers that need the + reference beyond one expression; each releases explicitly. + + Returns the raw CF reference WITHOUT interpreting it — callers type-check via + :func:`cf_is` before use. A non-zero error yields ``(None, err)`` and the + out-parameter is never dereferenced: on failure AX leaves it untouched, so + reading it would dereference uninitialised stack memory. + """ + if not elem: + return None, AX_ERR_CANNOT_COMPLETE + libs = _frameworks() + out = c_void_p() + with cf_string(name) as key: + if not key: + return None, AX_ERR_CANNOT_COMPLETE + err = int(libs.ax.AXUIElementCopyAttributeValue(elem, key, ctypes.byref(out))) + if err != AX_ERR_SUCCESS: + return None, err + return out.value, err + + +@contextmanager +def ax_owned_attr(elem: Any, name: str) -> Iterator["tuple[Any, int]"]: + """:func:`ax_attr` with the mandatory ``CFRelease`` — the form to use. + + A context manager for the same reason :func:`cf_string` is one: the Create Rule + makes the release obligatory, a tree walk performs thousands of these reads, and + a ``try/finally`` at every call site is a rule that gets forgotten exactly once + and then leaks ~400KB per walk (measured, before this existed). + + Note what is NOT safe to do with the yielded value: any CF object derived from + it (an array's elements, a nested element) is only valid while it is alive, so a + caller that needs to keep such a derivative must ``CFRetain`` it — which is what + :func:`ax_retained_elements` does. + """ + value, err = ax_attr(elem, name) + try: + yield value, err + finally: + if value: + _frameworks().cf.CFRelease(value) + + +def ax_str(elem: Any, name: str) -> str: + """Read an AX attribute as ``str``, or ``""``. + + Returns ``""`` for a missing attribute AND for a present-but-not-a-string + one. Both are ordinary: a window answers ``-25205`` for ``AXValue``, and + ``AXValue`` on a slider is a CFNumber. + + The value is decoded into a Python ``str`` and released before returning, so no + CF reference escapes. This is THE hot path of a tree walk (~5 calls per node). + """ + with ax_owned_attr(elem, name) as (value, err): + if err != AX_ERR_SUCCESS or value is None: + return "" + if not cf_is(value, type_ids().string): + return "" + return cf_string_value(value) + + +def ax_bool(elem: Any, name: str, *, default: bool = True) -> bool: + """Read an AX attribute as ``bool``, falling back to *default*. + + *default* matters: ``AXEnabled`` is unsupported on many elements (a window + answers ``-25205``), and rendering every such node as "(disabled)" would be + both wrong and noisy. Absent means unknown, and unknown reads as enabled. + """ + with ax_owned_attr(elem, name) as (value, err): + if err != AX_ERR_SUCCESS or value is None: + return default + parsed = cf_bool_value(value) + return default if parsed is None else parsed + + +def ax_bool_opt(elem: Any, name: str) -> "bool | None": + """Read a boolean AX attribute, distinguishing ABSENT (``None``) from ``False``. + + :func:`ax_bool` collapses "unsupported" into a caller-chosen default, which is + right for ``AXEnabled`` (absent means enabled) and wrong for the trait + attributes: ``AXSelected`` absent means "this element has no notion of + selection", while ``AXSelected=false`` means "selectable, not selected". A + renderer that printed "(not selected)" for every node lacking the attribute + would bury the tree in noise, so the tri-state has to survive the read. + """ + with ax_owned_attr(elem, name) as (value, err): + if err != AX_ERR_SUCCESS or value is None: + return None + return cf_bool_value(value) + + +def ax_is_settable(elem: Any, name: str) -> bool: + """True when *name* can be WRITTEN on *elem* (``AXUIElementIsAttributeSettable``). + + The only reliable "is this input editable?" signal. ``AXEnabled`` is about + interactivity, not writability: a read-only text field (a disabled form input, + a log pane, a computed cell) reports ``AXEnabled=true`` and a readable + ``AXValue``, and typing into it silently does nothing. Surfacing settability + lets the model pick the editable field on the first try instead of discovering + the read-only one by failing against it. + + Fails CLOSED (``False``) on any error: an unknowable attribute is treated as + not settable, so the trait is never advertised on speculation. + """ + if not elem: + return False + libs = _frameworks() + out = c_bool(False) + with cf_string(name) as cf_name: + if not cf_name: + return False + err = int(libs.ax.AXUIElementIsAttributeSettable(elem, cf_name, ctypes.byref(out))) + return err == AX_ERR_SUCCESS and bool(out.value) + + +def ax_parent(elem: Any) -> Any: + """The element's parent as a RETAINED reference, or ``None``. + + +1 like :func:`ax_retained_elements`, and for the same reason: the value comes + from a ``Copy`` call whose result the caller must own, and ``ax_owned_attr`` + releases it on scope exit — using it afterwards without the retain is a + use-after-free in another process's accessibility client. + + ``None`` for the root and for any element whose app does not answer + ``AXParent`` (ordinary), so callers treat it as "stop walking up". + """ + if not elem: + return None + with ax_owned_attr(elem, AX_PARENT) as (value, err): + if err != AX_ERR_SUCCESS or value is None: + return None + if not cf_is(value, type_ids().ax_element): + return None + return retain(value) + + +def same_element(a: Any, b: Any) -> bool: + """True when *a* and *b* name the SAME accessibility element. + + ``CFEqual``, not ``==``. ``AXFocusedUIElement`` returns a freshly created + reference, so the pointer it yields differs from the one the same node got + while being walked from ``AXChildren`` even though both address one UI element. + Comparing addresses would therefore report "not focused" for every element — + the focus marker would simply never appear, which is a silent no-op rather + than a visible failure. + + Both NULL is ``False``, not ``True``: "no element" is not identity with "no + element", and the one caller (the focus marker) must not light up every record + when nothing is focused. + """ + if not a or not b: + return False + return bool(_frameworks().cf.CFEqual(a, b)) + + +def ax_frame(elem: Any) -> "tuple[float, float, float, float] | None": + """``(x, y, width, height)`` of *elem* in TOP-LEFT SCREEN coordinates, or ``None``. + + ``AXPosition`` and ``AXSize`` come back as ``AXValue`` boxes rather than plain + numbers, so each is type-checked (``AXValueGetType``) before it is unboxed into + the matching struct — reading a ``CGPoint`` box into a ``CGSize`` buffer would + transpose ``y`` into ``width`` and produce a plausible-looking wrong rect, + which is the worst failure mode for a coordinate a click is aimed at. + + Returns ``None`` unless BOTH reads succeed. A partial frame is worse than no + frame: the caller would have to invent the missing half, and every consumer + (rendering, hit-testing, the click ladder) would then be reasoning about a + rectangle that does not exist. Many elements legitimately have no geometry + (a menu bar item of a background app, an off-screen row), so ``None`` is an + ordinary answer and not an error. + + Top-left screen coordinates: this is the AX convention AND the convention this + whole package uses, so no flip happens here. The window-relative conversion is + the caller's job, because only the caller knows which window the element + belongs to. + """ + if not elem: + return None + point = _ax_value_point(elem, AX_POSITION) + if point is None: + return None + size = _ax_value_size(elem, AX_SIZE) + if size is None: + return None + return (point[0], point[1], size[0], size[1]) + + +def _ax_value_point(elem: Any, name: str) -> "tuple[float, float] | None": + """Unbox a ``CGPoint``-typed ``AXValue`` attribute, or ``None``.""" + out = CGPoint() + if not _unbox_ax_value(elem, name, K_AX_VALUE_CG_POINT, out): + return None + return (float(out.x), float(out.y)) + + +def _ax_value_size(elem: Any, name: str) -> "tuple[float, float] | None": + """Unbox a ``CGSize``-typed ``AXValue`` attribute, or ``None``.""" + out = CGSize() + if not _unbox_ax_value(elem, name, K_AX_VALUE_CG_SIZE, out): + return None + return (float(out.width), float(out.height)) + + +def _unbox_ax_value(elem: Any, name: str, want_type: int, out: Any) -> bool: + """Read *name* off *elem* and unbox it into *out*, iff it really is *want_type*. + + Three checks, none of them optional: + + * the attribute read succeeded and returned something; + * the something is an ``AXValue`` (``cf_is`` — an app may answer with a + CFNumber or a string, and ``AXValueGetType`` on a non-``AXValue`` is + undefined behaviour, not an error return); + * the box's own type matches *want_type*, so the bytes land in the right + struct fields. + + ``AXValueGetValue`` also returns false when the type disagrees, but relying on + that alone would mean calling it on a pointer that may not be an ``AXValue`` + at all — the same class of crash ``cf_string_value``'s type guard exists to + prevent. + """ + with ax_owned_attr(elem, name) as (value, err): + if err != AX_ERR_SUCCESS or value is None: + return False + if not cf_is(value, type_ids().ax_value): + return False + libs = _frameworks() + if int(libs.ax.AXValueGetType(value)) != want_type: + return False + return bool(libs.ax.AXValueGetValue(value, want_type, ctypes.byref(out))) + + +def ax_children(elem: Any, *, limit: int = 0) -> list[Any]: + """The element's children as RETAINED references — caller must release. + + See :func:`ax_retained_elements` for why retaining is required rather than + optional. Every caller passes the result to :func:`release_all` when done; a + tree walk does so as it pops each node. + """ + return ax_retained_elements(elem, "AXChildren", limit=limit) + + +def ax_retained_elements(elem: Any, name: str, *, limit: int = 0) -> list[Any]: + """Read an element-array attribute (``AXChildren``, ``AXWindows``) as +1 refs. + + Two ownership rules collide here and both have to be honoured: + + * the ARRAY comes from a ``Copy`` call, so we own it and must release it; + * its ELEMENTS are owned by the array, so they die with it. + + A caller that wants to keep a child after the array is gone must therefore + ``CFRetain`` it. Verified live: a child read back its ``AXRole`` correctly after + the parent array was released *because it had been retained* — without the + retain that is a use-after-free in another process's accessibility client, which + is a crash rather than an exception. + + So this returns +1 references and the caller MUST call :func:`release_all`. + Each entry is also type-checked to be an ``AXUIElement``: a malformed app could + answer with an array of strings, and performing an action on a CFString pointer + is a segfault. + """ + libs = _frameworks() + with ax_owned_attr(elem, name) as (value, err): + if err != AX_ERR_SUCCESS: + return [] + element_type = type_ids().ax_element + return [ + libs.cf.CFRetain(item) + for item in cf_array_items(value, limit=limit) + if cf_is(item, element_type) + ] + + +def retain(ref: Any) -> Any: + """``CFRetain(ref)`` and return it — for a reference that must outlive its owner. + + Used when the walk hands one element back to its caller: the element belongs to + a parent array the walk is about to release, and using it afterwards without a + retain is a use-after-free in another process's accessibility client (a crash, + not an exception). + """ + if not ref: + return ref + return _frameworks().cf.CFRetain(ref) + + +def ax_attr_error(elem: Any, name: str) -> "tuple[bool, int]": + """``(present, ax_error)`` for an attribute, WITHOUT leaking the value. + + For the caller that only needs to distinguish *why* a read came back empty — + ``-25204`` ("Electron has not opted in, retry") versus a genuinely absent + attribute. Releasing here means the diagnostic path cannot become a leak of its + own. + """ + with ax_owned_attr(elem, name) as (value, err): + return bool(value), err + + +def release_all(refs: "Sequence[Any]") -> None: + """``CFRelease`` every non-NULL reference in *refs*. + + The counterpart to :func:`ax_retained_elements`. Kept as a named helper so the + release side of the walk reads as deliberately as the retain side. + """ + libs = _frameworks() + for ref in refs: + if ref: + libs.cf.CFRelease(ref) + + +def ax_actions(elem: Any, *, limit: int = 0) -> tuple[str, ...]: + """The element's advertised action names, SANITIZED. + + Advisory only. A real ``AXScrollArea`` advertised ``AXScrollDownByPage`` and + then returned ``-25205`` when asked to perform it, so every caller must try + the action, check the error and fall back. + + Sanitized at this boundary — see :func:`_sanitize_action` — because an action + name is arbitrary app-controlled text, not the ``AX*`` identifier one expects. + """ + if not elem: + return () + libs = _frameworks() + out = c_void_p() + err = int(libs.ax.AXUIElementCopyActionNames(elem, ctypes.byref(out))) + if err != AX_ERR_SUCCESS: + return () + try: + names = [ + _sanitize_action(cf_string_value(item)) + for item in cf_array_items(out.value, limit=limit) + ] + finally: + # ``Copy`` in the name => Create Rule => we own the array. The name strings + # are decoded into Python above and belong to the array, so releasing it + # here is the complete cleanup. + if out.value: + libs.cf.CFRelease(out.value) + return tuple(name for name in names if name) + + +# Bounds for a sanitized action name. Real identifiers are short (``AXPress``, +# ``AXShowMenu``); the cap only bites on the app-generated descriptions below. +MAX_ACTION_NAME_LEN = 48 + + +def _sanitize_action(name: str) -> str: + """Collapse whitespace in an action name and clip it. + + **A live-data finding, and a real injection vector.** Action names are NOT + guaranteed to be ``AX*`` identifiers — macOS Notes advertises + ``'Name:remove pin\\nTarget:0x0\\nSelector:(null)'`` on its table rows, i.e. an + app-generated description containing EMBEDDED NEWLINES. + + The rendered tree's structure IS its indentation, and the renderer joins action + names into the element's line. An embedded newline therefore lets app content + forge additional tree lines — fabricating elements with indices the model would + then try to address. Values and titles are already collapsed when rendered; + actions were not, and the safest place to fix it is here at the boundary that + produces the data, so no downstream renderer has to remember. + + Clipped as well as collapsed: a 200-character action description crowds out + the rest of the line for no informational gain. + """ + flat = " ".join(name.split()) + return flat[:MAX_ACTION_NAME_LEN] if len(flat) > MAX_ACTION_NAME_LEN else flat + + +def ax_perform(elem: Any, action: str) -> int: + """Perform a named accessibility action. Returns the raw AX error code. + + The RAW code is returned rather than a bool because the callers branch on it: + ``-25205`` means "advertised but unsupported — use the fallback", while other + codes are genuine failures whose number belongs in the error message so a + support thread is diagnosable. + """ + if not elem: + return AX_ERR_CANNOT_COMPLETE + with cf_string(action) as name: + if not name: + return AX_ERR_CANNOT_COMPLETE + return int(_frameworks().ax.AXUIElementPerformAction(elem, name)) + + +def ax_set_str(elem: Any, name: str, value: str) -> int: + """Set a string-valued AX attribute (the ``set_value`` primitive).""" + if not elem: + return AX_ERR_CANNOT_COMPLETE + with cf_string(name) as key: + if not key: + return AX_ERR_CANNOT_COMPLETE + with cf_string(value) as payload: + if not payload: + return AX_ERR_CANNOT_COMPLETE + return int(_frameworks().ax.AXUIElementSetAttributeValue(elem, key, payload)) + + +def ax_set_true(elem: Any, name: str) -> int: + """Set a boolean AX attribute to ``kCFBooleanTrue``. + + Used for ``AXManualAccessibility`` (the Electron opt-in) and ``AXFocused`` + (so typed keystrokes land in the addressed field rather than wherever the + target app last had focus). + """ + if not elem: + return AX_ERR_CANNOT_COMPLETE + with cf_string(name) as key: + if not key: + return AX_ERR_CANNOT_COMPLETE + return int(_frameworks().ax.AXUIElementSetAttributeValue(elem, key, cf_true())) + + +def ax_is_process_trusted() -> bool: + """``AXIsProcessTrusted()`` — ADVISORY. See :mod:`permissions`.""" + return bool(_frameworks().ax.AXIsProcessTrusted()) + + +def ax_window_id(window: Any) -> int: + """The CoreGraphics window id for an AX window element, or ``0``. + + Correlating the two window namespaces is a CORRECTNESS requirement. + ``AXWindows`` and the CoreGraphics window list are independently ordered: + verified live on Notes, ``AXWindows[0]`` was an ``AXDialog`` with an empty + title exposing 3 nodes, while the frontmost CG window was the real 103-node + document window. Walking one and capturing the other would hand the model a + tree and a screenshot of DIFFERENT WINDOWS — a silent, confidently-wrong + observation, which is worse than an error. + + Returns ``0`` when the private symbol is unavailable or the element is not a + window (AX answers ``-25201`` for the application element, verified). Callers + fall back to matching on title/``AXMain``. + """ + libs = _frameworks() + getter = getattr(libs.ax, "_AXUIElementGetWindow", None) + if getter is None or not window: + return 0 + out = c_uint32(0) + if int(getter(window, ctypes.byref(out))) != AX_ERR_SUCCESS: + return 0 + return int(out.value) + + +# ── CoreGraphics: window list ── + + +def window_list() -> list[WindowInfo]: + """Enumerate on-screen windows, excluding desktop elements. + + This list is the ONLY app/pid resolution mechanism in the package. A + process-name search returns short-lived helper processes: ``pgrep -n + "Google Chrome"`` answered 47492 (a helper that answered ``-25204`` to every + attribute read and then exited) while the real browser was 637, and Slack's + helper 1614 shadowed the real 942. The pid that owns a visible, layer-0 + window is the one whose accessibility tree is populated. + """ + libs = _frameworks() + info = c_void_p( + libs.cg.CGWindowListCopyWindowInfo( + K_CG_WINDOW_LIST_ON_SCREEN_ONLY | K_CG_WINDOW_LIST_EXCLUDE_DESKTOP, + K_CG_NULL_WINDOW_ID, + ) + ) + if not info: + return [] + try: + out: list[WindowInfo] = [] + dict_type = type_ids().dictionary + for entry in cf_array_items(info): + if not cf_is(entry, dict_type): + continue + window_id = _dict_int(entry, CG_WINDOW_NUMBER) + pid = _dict_int(entry, CG_WINDOW_OWNER_PID) + if window_id is None or pid is None: + continue + out.append( + WindowInfo( + window_id=window_id, + pid=pid, + owner_name=_dict_str(entry, CG_WINDOW_OWNER_NAME), + title=_dict_str(entry, CG_WINDOW_NAME), + layer=_dict_int(entry, CG_WINDOW_LAYER) or 0, + bounds=_dict_bounds(entry), + ) + ) + return out + finally: + # The window list is a "Copy" API: we own the array and must release it, + # even if a malformed entry raised on the way through. + libs.cf.CFRelease(info) + + +def _dict_int(entry: Any, key: str) -> "int | None": + """Read an int out of a window-list dictionary.""" + with cf_string(key) as ref: + if not ref: + return None + value = _frameworks().cf.CFDictionaryGetValue(entry, ref) + return cf_number_int(value) + + +def _dict_str(entry: Any, key: str) -> str: + """Read a string out of a window-list dictionary.""" + with cf_string(key) as ref: + if not ref: + return "" + value = _frameworks().cf.CFDictionaryGetValue(entry, ref) + if not cf_is(value, type_ids().string): + return "" + return cf_string_value(value) + + +def _dict_float(entry: Any, key: str) -> "float | None": + """Read a float out of a CFDictionary (the bounds sub-dictionary).""" + with cf_string(key) as ref: + if not ref: + return None + value = _frameworks().cf.CFDictionaryGetValue(entry, ref) + return cf_number_double(value) + + +def _dict_bounds(entry: Any) -> "tuple[float, float, float, float] | None": + """Read ``kCGWindowBounds`` as ``(x, y, w, h)`` in TOP-LEFT coordinates. + + Returns ``None`` — never a partial or zero rect — when the key is missing or + any component is unreadable. A caller uses this to decide whether a screen + point belongs to an authorized application, so "unknown" must be + distinguishable from "a rect at the origin": the latter would silently claim + the top-left corner of the display. + """ + with cf_string(CG_WINDOW_BOUNDS) as ref: + if not ref: + return None + raw = _frameworks().cf.CFDictionaryGetValue(entry, ref) + if not cf_is(raw, type_ids().dictionary): + return None + x = _dict_float(raw, CG_BOUNDS_X) + y = _dict_float(raw, CG_BOUNDS_Y) + width = _dict_float(raw, CG_BOUNDS_W) + height = _dict_float(raw, CG_BOUNDS_H) + # Read into four names rather than a comprehension so the None-checks actually + # narrow the types (a list comprehension keeps them ``float | None`` no matter + # what the ``any(... is None)`` guard proved). + if x is None or y is None or width is None or height is None: + return None + if width <= 0.0 or height <= 0.0: + return None + return (x, y, width, height) + + +def executable_path(pid: int) -> str: + """Absolute executable path of *pid* via ``proc_pidpath``, or ``""``. + + ``proc_pidpath`` rather than reading ``/proc`` (which macOS does not have) or + shelling out to ``ps``: it is a single syscall, needs no subprocess, and + measured under 1ms. Returns ``""`` for a dead pid or a process we cannot + inspect — never raises, because this feeds identity resolution, and a failure + there must produce "identity unknown" (which the governance gate DENIES), not + an exception. + """ + libs = _frameworks() + buf = ctypes.create_string_buffer(PROC_PIDPATH_MAX) + written = int(libs.proc.proc_pidpath(int(pid), buf, PROC_PIDPATH_MAX)) + if written <= 0: + return "" + return buf.value.decode("utf-8", "replace") + + +# ── CoreGraphics: event synthesis ── + + +def event_source() -> c_void_p: + """The cached PRIVATE ``CGEventSource``. + + ``kCGEventSourceStatePrivate`` is load-bearing, not a preference. Events + built from the default (HID) source inherit the user's LIVE modifier state: + asking a live prototype to type ``abc`` produced ``' I Abc'`` because the + user happened to be holding keys. A private source starts from a clean + modifier state that only :func:`post_key`'s explicit flags populate. + + Cached process-wide: creating one per keystroke would be both wasteful and a + leak risk in the middle of a loop. + """ + global _event_source + libs = _frameworks() + if _event_source is not None: + return _event_source + with _init_lock: + if _event_source is None: + _event_source = c_void_p(libs.cg.CGEventSourceCreate(K_CG_EVENT_SOURCE_STATE_PRIVATE)) + return _event_source + + +def post_key(pid: int, keycode: int, flags: int = 0) -> None: + """Post one key-down/key-up pair for *keycode* to *pid*. + + Three rules, all verified load-bearing and all violated by the obvious + implementation: + + 1. the event source is PRIVATE (:func:`event_source`); + 2. ``CGEventSetFlags`` is called on EVERY event **including when flags is + 0** — that explicit zero is what clears the inherited modifier state, so + skipping it "because there are no modifiers" reintroduces the bug; + 3. delivery is ``CGEventPostToPid``, never ``CGEventPost``. The latter goes + to the global event tap, i.e. to whatever window the user is actually + typing in — a keystroke aimed at a background app would land in their + editor. + """ + libs = _frameworks() + source = event_source() + for is_down in (True, False): + event = c_void_p( + libs.cg.CGEventCreateKeyboardEvent(source, c_uint16(int(keycode)), c_bool(is_down)) + ) + if not event: + continue + try: + libs.cg.CGEventSetFlags(event, c_uint64(int(flags))) + libs.cg.CGEventPostToPid(int(pid), event) + finally: + libs.cf.CFRelease(event) + + +def post_text(pid: int, text: str) -> None: + """Type *text* into *pid* as unicode key events. + + Uses ``CGEventKeyboardSetUnicodeString`` with a keycode of 0 rather than a + per-character keycode lookup: that makes the path layout-independent and able + to emit characters the US layout cannot reach in one keystroke (verified: a + round-trip of ``"Hello, KiroCrew! aA$ 123"`` came back byte-identical). Flags + are still set explicitly to zero on every event — the unicode payload + replaces the *keystroke*, not the modifier hygiene. + + Encoded per character as UTF-16LE so an astral character is delivered as its + full surrogate pair in one event rather than as two lone surrogates. + """ + libs = _frameworks() + source = event_source() + for char in text: + units = char.encode("utf-16-le") + count = len(units) // 2 + if count <= 0: + continue + buf = (c_uint16 * count).from_buffer_copy(units) + for is_down in (True, False): + event = c_void_p( + libs.cg.CGEventCreateKeyboardEvent(source, c_uint16(0), c_bool(is_down)) + ) + if not event: + continue + try: + libs.cg.CGEventSetFlags(event, c_uint64(0)) + libs.cg.CGEventKeyboardSetUnicodeString(event, c_long(count), buf) + libs.cg.CGEventPostToPid(int(pid), event) + finally: + libs.cf.CFRelease(event) + + +def post_scroll(pid: int, delta_y: int, delta_x: int) -> None: + """Post one scroll-wheel event to *pid*, in LINE units. + + **Never uses the multi-axis constructor.** + ``CGEventCreateScrollWheelEvent`` is variadic, and ctypes marshals a variadic + tail incorrectly on arm64: declaring the 5-argument form and asking for + ``(axis1=-3, axis2=0)`` produced ``axis2=30416`` — an arbitrary sideways + scroll of a live window. So the event is built with ``wheelCount=1`` through + the FIXED prototype and both axes are then written with the non-variadic + ``CGEventSetIntegerValueField``, which was verified exact for every input. + """ + libs = _frameworks() + source = event_source() + event = c_void_p( + libs.cg.CGEventCreateScrollWheelEvent(source, K_CG_SCROLL_EVENT_UNIT_LINE, 1, c_int32(0)) + ) + if not event: + return + try: + libs.cg.CGEventSetIntegerValueField(event, K_CG_SCROLL_DELTA_AXIS_1, c_int64(int(delta_y))) + libs.cg.CGEventSetIntegerValueField(event, K_CG_SCROLL_DELTA_AXIS_2, c_int64(int(delta_x))) + libs.cg.CGEventSetFlags(event, c_uint64(0)) + libs.cg.CGEventPostToPid(int(pid), event) + finally: + libs.cf.CFRelease(event) + + +def mouse_button_codes(button: str) -> tuple[int, int, int, int]: + """``(button_number, down_type, up_type, dragged_type)`` for *button*. + + Raises :class:`ComputerUseUnsupported` for an unknown name rather than + defaulting to left: silently turning an unrecognised ``mouse_button`` into a + left click would send a DIFFERENT gesture than the caller asked for into a live + application, which is the same class of defect as + :func:`~kiro_crew.computer_use.keymap.parse_key` refusing an unknown modifier. + The dispatch layer validates the enum first, so this is the belt for the + in-process entry point. + """ + codes = MOUSE_EVENT_TYPES.get(button) + if codes is None: + raise ComputerUseUnsupported(f"unknown mouse button {button!r}") + return codes + + +def _mouse_event( + libs: Libs, + event_type: int, + x: float, + y: float, + button_number: int, + *, + click_state: int = 0, +) -> "c_void_p | None": + """Create one mouse event, or ``None``. **Caller must ``CFRelease``.** + + Built from the PRIVATE event source for the same reason every keyboard event + is (:func:`event_source`): the default HID source inherits the user's live + modifier state, so a synthesized click would arrive as cmd-click if they + happened to be holding a key. Flags are set to an explicit zero here too — + the modifier hygiene is about the SOURCE of the event, not about whether it is + a keystroke. + + ``click_state`` is written only when non-zero; a double click is a pair whose + ``kCGMouseEventClickState`` is 2, NOT two separate pairs (AppKit reads + ``NSEvent.clickCount`` from that field). + """ + event = c_void_p( + libs.cg.CGEventCreateMouseEvent( + event_source(), + c_uint32(int(event_type)), + CGPoint(float(x), float(y)), + c_uint32(int(button_number)), + ) + ) + if not event: + return None + libs.cg.CGEventSetFlags(event, c_uint64(0)) + if click_state: + libs.cg.CGEventSetIntegerValueField( + event, K_CG_MOUSE_EVENT_CLICK_STATE, c_int64(int(click_state)) + ) + return event + + +def post_mouse_click( + pid: int, + x: float, + y: float, + *, + button: str = MOUSE_BUTTON_LEFT, + count: int = 1, +) -> None: + """Post *count* clicks at ``(x, y)`` to *pid*. **The pointer does NOT move.** + + This is the ``app_post`` click method: the event carries a location and is + delivered with ``CGEventPostToPid``, so the target application sees a click at + that point while the operator's physical cursor stays exactly where they left + it (verified live — the prototype's ``mouse_pos()`` was identical before and + after). That property is the whole reason this method exists and is the + default for a coordinate click; :func:`post_mouse_global` is the opt-in + alternative that does move it. + + ``count`` is expressed as the events' CLICK STATE, not as repeated pairs — see + :func:`_mouse_event`. Each pair is still posted so an app that tracks + down/up transitions sees them, with a sub-double-click-interval pause between + them so the OS recognizes the repetition. + """ + libs = _frameworks() + button_number, down_type, up_type, _dragged = mouse_button_codes(button) + for click_index in range(max(1, int(count))): + click_state = click_index + 1 + for event_type in (down_type, up_type): + event = _mouse_event(libs, event_type, x, y, button_number, click_state=click_state) + if event is None: + continue + try: + libs.cg.CGEventPostToPid(int(pid), event) + finally: + libs.cf.CFRelease(event) + time.sleep(CLICK_PAIR_DELAY_SECS) + + +def post_mouse_drag( + pid: int, + start: tuple[float, float], + end: tuple[float, float], + *, + button: str = MOUSE_BUTTON_LEFT, + steps: int = DRAG_STEPS, +) -> None: + """Drag from *start* to *end* inside *pid*. **The pointer does NOT move.** + + Down at the start, :data:`DRAG_STEPS` interpolated ``MouseDragged`` events, up + at the end — all app-targeted. The intermediate events are NOT padding: a bare + down/up pair is not a drag to most applications, because the gesture is + recognized from the motion between the endpoints (a canvas records two + isolated points and a text view selects nothing). Verified against TextEdit, + which needed the interpolation before it registered a selection. + + The per-step sleep matters for the same reason: identical timestamps let + AppKit's recognizer coalesce the whole sequence into one motion. + """ + libs = _frameworks() + button_number, down_type, up_type, dragged_type = mouse_button_codes(button) + x0, y0 = float(start[0]), float(start[1]) + x1, y1 = float(end[0]), float(end[1]) + total = max(1, int(steps)) + plan: list[tuple[int, float, float]] = [(down_type, x0, y0)] + for index in range(1, total): + ratio = index / total + plan.append((dragged_type, x0 + (x1 - x0) * ratio, y0 + (y1 - y0) * ratio)) + plan.append((up_type, x1, y1)) + for event_type, px, py in plan: + event = _mouse_event(libs, event_type, px, py, button_number) + if event is None: + continue + try: + libs.cg.CGEventPostToPid(int(pid), event) + finally: + libs.cf.CFRelease(event) + time.sleep(DRAG_STEP_DELAY_SECS) + + +def post_mouse_global( + x: float, + y: float, + *, + button: str = MOUSE_BUTTON_LEFT, + count: int = 1, +) -> None: + """MOVE THE OPERATOR'S REAL POINTER to ``(x, y)`` and click it there. + + **The one function in this module that takes over the physical mouse**, and + the only caller of ``CGWarpMouseCursorPosition`` and ``CGEventPost``. Every + other input path is app-scoped and leaves the cursor untouched. + + Reachable only through ``click_method: "global"``, which the model must NAME — + ``policy.resolve_click_method`` never resolves ``auto`` onto it. Do not add a + second caller: that resolution happens at the dispatch chokepoint, so a call + site that bypasses the chokepoint bypasses the naming requirement too. + + Why it exists at all, given the app-scoped path works: some UI is only + reachable by a real physical click — a Dock item, a menu-bar extra, a window + belonging to a process whose event queue refuses posted events, a Space + switcher. The warp is what makes the subsequent global click land where the + caller asked, because a global event is delivered to whatever is under the + cursor rather than to a pid. + + ``CGEventPost`` to ``kCGHIDEventTap`` rather than ``CGEventPostToPid``: there + is no per-pid form of "simulate a physical mouse", and delivering to a pid + would defeat the point (the app-scoped path already does that, better). + """ + libs = _frameworks() + button_number, down_type, up_type, _dragged = mouse_button_codes(button) + # Warp FIRST: a global click is delivered to whatever sits under the cursor, so + # posting before the warp would click wherever the operator happened to leave + # it. The return code is advisory — a refused warp still leaves a coherent + # click at the (unchanged) cursor, and refusing the whole action would be worse + # than a click the caller can verify from the refreshed tree. + warped = int(libs.cg.CGWarpMouseCursorPosition(CGPoint(float(x), float(y)))) + if warped != 0: + logger.debug("CGWarpMouseCursorPosition returned %s for (%s, %s)", warped, x, y) + for click_index in range(max(1, int(count))): + click_state = click_index + 1 + for event_type in (down_type, up_type): + event = _mouse_event(libs, event_type, x, y, button_number, click_state=click_state) + if event is None: + continue + try: + libs.cg.CGEventPost(c_uint32(K_CG_HID_EVENT_TAP), event) + finally: + libs.cf.CFRelease(event) + time.sleep(CLICK_PAIR_DELAY_SECS) + + +def post_mouse_drag_global( + start: tuple[float, float], + end: tuple[float, float], + *, + button: str = MOUSE_BUTTON_LEFT, + steps: int = DRAG_STEPS, +) -> None: + """MOVE THE OPERATOR'S REAL POINTER along a drag from *start* to *end*. + + The pointer-moving counterpart of :func:`post_mouse_drag`, subject to the + identical two permits (see :func:`post_mouse_global`). The cursor is warped + before EVERY event, not only the first: a global drag is only coherent if the + physical cursor tracks the synthesized motion, and warping once would leave + the OS delivering the intermediate events to whatever sat under the start + point. + """ + libs = _frameworks() + button_number, down_type, up_type, dragged_type = mouse_button_codes(button) + x0, y0 = float(start[0]), float(start[1]) + x1, y1 = float(end[0]), float(end[1]) + total = max(1, int(steps)) + plan: list[tuple[int, float, float]] = [(down_type, x0, y0)] + for index in range(1, total): + ratio = index / total + plan.append((dragged_type, x0 + (x1 - x0) * ratio, y0 + (y1 - y0) * ratio)) + plan.append((up_type, x1, y1)) + for event_type, px, py in plan: + libs.cg.CGWarpMouseCursorPosition(CGPoint(px, py)) + event = _mouse_event(libs, event_type, px, py, button_number) + if event is None: + continue + try: + libs.cg.CGEventPost(c_uint32(K_CG_HID_EVENT_TAP), event) + finally: + libs.cf.CFRelease(event) + time.sleep(DRAG_STEP_DELAY_SECS) + + +# ── CoreGraphics + ImageIO: in-process window capture ── + + +def preflight_screen_capture() -> bool: + """``CGPreflightScreenCaptureAccess()`` — ADVISORY. See :mod:`permissions`. + + The request variant is deliberately not bound: it pops a system dialog from + the calling process, which from a background sidecar is an unattributable + prompt. + """ + return bool(_frameworks().cg.CGPreflightScreenCaptureAccess()) + + +def capture_window_jpeg(window_id: int, *, max_px: int, quality: float) -> tuple[bytes, int, int]: + """Capture one window's pixels and JPEG-encode them, entirely in-process. + + Returns ``(jpeg_bytes, width, height)``, or ``(b"", 0, 0)`` when the window + cannot be captured (a closed or invalid window id yields a NULL image — + verified, not a crash). ImageIO performs the downscale itself via + ``kCGImageDestinationImageMaxPixelSize``, so there is **no subprocess and no + image library**: no ``screencapture``, no Pillow, nothing for the spawn audit + to account for and no optional dependency to degrade around. + + A null ``CGRect`` (all zeros) means "the window's own bounds", which is what + keeps the capture scoped to ONE window rather than to the screen. + + Every CF/CG object is released in a ``finally`` — **including when + ``Finalize`` returns False.** That branch is the easy one to get wrong: an + early return on a failed encode is exactly when four objects leak, and it is + also the branch a test exercises. + """ + libs = _frameworks() + image = c_void_p( + libs.cg.CGWindowListCreateImage( + CGRect(CGPoint(0.0, 0.0), CGSize(0.0, 0.0)), + K_CG_WINDOW_LIST_INCLUDING_WINDOW, + c_uint32(int(window_id)), + K_CG_WINDOW_IMAGE_BOUNDS_IGNORE_FRAMING, + ) + ) + if not image: + return b"", 0, 0 + + data: "c_void_p | None" = None + dest: "c_void_p | None" = None + options: "c_void_p | None" = None + num_max: "c_void_p | None" = None + num_quality: "c_void_p | None" = None + try: + data = c_void_p(libs.cf.CFDataCreateMutable(None, 0)) + if not data: + return b"", 0, 0 + with cf_string(UTI_JPEG) as uti: + if not uti: + return b"", 0, 0 + dest = c_void_p(libs.io.CGImageDestinationCreateWithData(data, uti, 1, None)) + if not dest: + return b"", 0, 0 + + options = _cf_type_dict() + if not options: + return b"", 0, 0 + max_value = c_int32(int(max_px)) + num_max = c_void_p( + libs.cf.CFNumberCreate(None, K_CF_NUMBER_SINT32, ctypes.byref(max_value)) + ) + quality_value = c_double(float(quality)) + num_quality = c_void_p( + libs.cf.CFNumberCreate(None, K_CF_NUMBER_DOUBLE, ctypes.byref(quality_value)) + ) + max_key = c_void_p.in_dll(libs.io, IO_KEY_MAX_PIXEL_SIZE) + quality_key = c_void_p.in_dll(libs.io, IO_KEY_LOSSY_QUALITY) + if num_max: + libs.cf.CFDictionarySetValue(options, max_key, num_max) + if num_quality: + libs.cf.CFDictionarySetValue(options, quality_key, num_quality) + + libs.io.CGImageDestinationAddImage(dest, image, options) + if not libs.io.CGImageDestinationFinalize(dest): + # Degrade to "no image": the accessibility tree is the primary + # channel, so a failed encode must not fail the whole observation. + logger.debug("ImageIO finalize failed for window %s", window_id) + return b"", 0, 0 + length = int(libs.cf.CFDataGetLength(data)) + if length <= 0: + return b"", 0, 0 + raw = ctypes.string_at(libs.cf.CFDataGetBytePtr(data), length) + width, height = jpeg_dimensions(raw) + return raw, width, height + finally: + for obj in (num_max, num_quality, options, dest, data): + if obj: + libs.cf.CFRelease(obj) + libs.cg.CGImageRelease(image) + + +# JPEG marker bytes for :func:`jpeg_dimensions`. +_JPEG_SOI = 0xD8 +_JPEG_EOI = 0xD9 +_JPEG_MARKER = 0xFF +_JPEG_SOF_MARKERS = frozenset({0xC0, 0xC1, 0xC2, 0xC3, 0xC5, 0xC6, 0xC7, 0xC9, 0xCA, 0xCB}) +_JPEG_RST_FIRST = 0xD0 +_JPEG_RST_LAST = 0xD7 +_JPEG_SOF_DIMENSION_OFFSET = 5 + + +def jpeg_dimensions(raw: bytes) -> tuple[int, int]: + """Parse ``(width, height)` out of JPEG bytes, or ``(0, 0)``. + + The ENCODED dimensions are what we report, not the source image's: + ``kCGImageDestinationImageMaxPixelSize`` downscales inside ImageIO, so + ``CGImageGetWidth`` on the input would over-report by the scale factor and + the size line shown to the model would be wrong (verified: a 1676x1320 window + encoded to 1280x1008). + + A ~25-line stdlib scan rather than an image library: this is the only reason + a dependency would be needed at all, and reporting ``(0, 0)`` on an + unparseable stream is a cosmetic degradation, never a failure. + """ + if len(raw) < 4 or raw[0] != _JPEG_MARKER or raw[1] != _JPEG_SOI: + return 0, 0 + pos = 2 + while pos + 3 < len(raw): + if raw[pos] != _JPEG_MARKER: + pos += 1 + continue + marker = raw[pos + 1] + if marker in (_JPEG_MARKER, _JPEG_SOI, _JPEG_EOI) or ( + _JPEG_RST_FIRST <= marker <= _JPEG_RST_LAST + ): + pos += 2 + continue + if marker in _JPEG_SOF_MARKERS: + start = pos + _JPEG_SOF_DIMENSION_OFFSET + if start + 4 > len(raw): + return 0, 0 + height, width = struct.unpack(">HH", raw[start : start + 4]) + return int(width), int(height) + segment = struct.unpack(">H", raw[pos + 2 : pos + 4])[0] + if segment < 2: + return 0, 0 + pos += 2 + segment + return 0, 0 + + +def shots_dir_default() -> str: + """Default screenshot directory path (no side effects). + + ``tempfile.gettempdir()`` rather than a hardcoded ``/tmp``: it honours + ``$TMPDIR`` on POSIX and resolves to ``%TEMP%`` on Windows, where ``/tmp`` + does not exist. Kept here beside the capture primitive so the path and the + encoder cannot drift apart; :mod:`capture_macos` owns creating it with the + right mode. + """ + return os.path.join(tempfile.gettempdir(), SCREENSHOT_DIR_NAME) + + +__all__ = [ + "AX_ERR_ACTION_UNSUPPORTED", + "AX_ERR_ATTRIBUTE_UNSUPPORTED", + "AX_ERR_CANNOT_COMPLETE", + "AX_ERR_NO_VALUE", + "AX_ERR_SUCCESS", + "AX_MANUAL_ACCESSIBILITY", + "AX_MESSAGING_TIMEOUT_SECS", + "CGPoint", + "CGRect", + "CGSize", + "CLICK_PAIR_DELAY_SECS", + "DRAG_STEPS", + "DRAG_STEP_DELAY_SECS", + "K_CG_HID_EVENT_TAP", + "K_CG_MOUSE_EVENT_CLICK_STATE", + "MOUSE_EVENT_TYPES", + "ELECTRON_OPT_IN_POLL_SECS", + "ELECTRON_OPT_IN_WAIT_SECS", + "CG_WINDOW_LAYER_NORMAL", + "Libs", + "TypeIds", + "WindowInfo", + "available", + "ax_actions", + "ax_app_element", + "ax_application", + "ax_attr", + "ax_attr_error", + "ax_bool", + "ax_children", + "ax_is_process_trusted", + "ax_owned_attr", + "ax_perform", + "ax_retained_elements", + "ax_set_str", + "ax_set_true", + "ax_str", + "ax_window_id", + "capture_window_jpeg", + "cf_array_items", + "cf_bool_value", + "cf_is", + "cf_number_int", + "cf_string", + "cf_string_value", + "cf_true", + "event_source", + "executable_path", + "frameworks", + "jpeg_dimensions", + "mouse_button_codes", + "post_key", + "post_mouse_click", + "post_mouse_drag", + "post_mouse_drag_global", + "post_mouse_global", + "post_scroll", + "post_text", + "preflight_screen_capture", + "release_all", + "reset_frameworks", + "retain", + "shots_dir_default", + "type_ids", + "window_list", +] diff --git a/src/kiro_crew/computer_use/macos_skylight.py b/src/kiro_crew/computer_use/macos_skylight.py new file mode 100644 index 00000000000..c5b97391b65 --- /dev/null +++ b/src/kiro_crew/computer_use/macos_skylight.py @@ -0,0 +1,669 @@ +"""``click_method: "sky_click"`` — the PRIVATE SkyLight background-window click. + +**This module is quarantine.** It is the ONLY file in the package allowed to touch +undocumented Apple ABI, and it exists as its own module rather than living in +``macos_ffi.py`` so that: + +* ``macos_ffi.py`` keeps its "public frameworks only" property, which is what makes + it reviewable against Apple's documentation; +* every future macOS-compatibility break has ONE review boundary — if a point + release changes a byte offset or drops a symbol, the blast radius is this file; +* the private surface can be disabled wholesale (see :func:`available`) without + touching any other code path. + +Other implementations of this technique isolate it the same way; at least one ships +it in a separate signed helper process rather than inside the main binary, which is +the same instinct one level further. Worth revisiting if this surface grows. + +**What it buys.** ``sky_click`` clicks a window that is BEHIND other windows, +without raising it and without moving the operator's pointer. Neither public path +can do that: ``accessibility`` needs an addressable element (and many canvases have +none), and ``app_post`` delivers to the app but is ignored by renderers that +hit-test against the window server's idea of which window is in front — the +browser-engine-backed apps, in practice. That +gap is real and was hit in practice: a Freeform canvas behind a Zoom annotation +overlay could not be drawn on by any public method. + +**Where the ABI comes from.** The symbol declarations and the event-field recipe are +derived from prior permissively-licensed open-source work that reverse-engineered +this path (attributed in ``NOTICE``). None of it is published by +Apple: the byte layout of the activation record and the numeric event fields are +observed behaviour, and ``WindowServer`` does not document the accepted width of the +click-group field. Treat every constant here as observed, not specified. + +**Fail-closed, always.** :func:`available` reports which symbols are missing and +every entry point raises a typed, model-readable refusal when the SPI is not +usable. A future macOS that removes ``SLEventPostToPid`` must degrade to "this +method is unavailable, use app_post" — never to a crash, and never to a silently +mis-delivered click. +""" + +from __future__ import annotations + +import ctypes +import ctypes.util +import logging +import threading +import time +from ctypes import ( + POINTER, + c_double, + c_int32, + c_int64, + c_uint8, + c_uint32, + c_void_p, +) +from dataclasses import dataclass, field +from typing import Any + +from kiro_crew import platform_compat +from kiro_crew.computer_use import macos_ffi +from kiro_crew.computer_use.types import ERR_SKY_CLICK_BUTTON +from kiro_crew.computer_use.types import MOUSE_BUTTON_LEFT as _LEFT_BUTTON +from kiro_crew.computer_use.types import ComputerUseError + +logger = logging.getLogger(__name__) + +# ── The private frameworks ── +# Absolute paths, not ``find_library``: these are PrivateFrameworks and are not on +# the standard search path. ``dlopen`` of an absent path simply fails, which is the +# degradation we want. +_SKYLIGHT_PATH = "/System/Library/PrivateFrameworks/SkyLight.framework/SkyLight" +_APP_SERVICES_PATH = ( + "/System/Library/Frameworks/ApplicationServices.framework/ApplicationServices" +) + +# ── The private symbols ── +# ``CGEventSetWindowLocation`` lives in SkyLight despite the CG prefix, and +# ``GetProcessForPID`` is a deprecated-but-present Carbon call in +# ApplicationServices. Named as constants so the "which symbol is missing" +# diagnostic can quote them back to the operator. +_SYM_POST_TO_PID = "SLEventPostToPid" +_SYM_SET_INT_FIELD = "SLEventSetIntegerValueField" +_SYM_SET_WINDOW_LOCATION = "CGEventSetWindowLocation" +_SYM_POST_EVENT_RECORD = "SLPSPostEventRecordTo" +_SYM_GET_PROCESS_FOR_PID = "GetProcessForPID" + +# ── The activation record ── +# A 0xF8-byte event record that tells the window server to treat one window of one +# process as synthetically focused. Every offset below is OBSERVED behaviour copied +# from the reference; none is documented. Apple can change any of it. +_RECORD_SIZE = 0xF8 +_RECORD_OFF_SIZE = 0x04 # carries the record's own length +_RECORD_OFF_KIND = 0x08 # 0x0D = the activation/deactivation kind +_RECORD_OFF_WINDOW_ID = 0x3C # 4 bytes, little-endian +_RECORD_OFF_FOCUSED = 0x8A # 0x01 = activate, 0x02 = deactivate +_RECORD_KIND_ACTIVATE = 0x0D +_RECORD_FOCUSED_YES = 0x01 +_RECORD_FOCUSED_NO = 0x02 +# A ProcessSerialNumber is two 32-bit halves; ``GetProcessForPID`` fills both. +_PSN_SIZE = 8 + +# ── The private event fields ── +# Raw ``CGEventField`` numbers the window server reads when routing a click to a +# specific window. The public enum does not name them. +_FIELD_GESTURE_PHASE = 0 +_FIELD_CLICK_STATE = 1 +_FIELD_BUTTON_NUMBER = 3 +_FIELD_SUBTYPE = 7 +_FIELD_TARGET_PID = 40 +_FIELD_WINDOW_NUMBER = 51 +_FIELD_CLICK_GROUP_ID = 58 +_FIELD_WINDOW_UNDER_POINTER = 91 +_FIELD_HANDLING_WINDOW_UNDER_POINTER = 92 +# Observed constant: subtype 3 is what a real windowed mouse event carries. +_EVENT_SUBTYPE_WINDOWED = 3 + +# ── The click recipe ── +# A "primer" down/up pair at (-1, -1) precedes the real click. It is not a retry: +# it is what makes the window server accept the following pair as a click on the +# TARGET window rather than on whatever is frontmost. Removing it makes the real +# click land on the front window, which is the exact bug this method exists to +# avoid. +_PRIMER_POINT = (-1.0, -1.0) +_PHASE_MOVED = 2 +_PHASE_PRIMER_DOWN = 1 +_PHASE_PRIMER_UP = 2 +_PHASE_TARGET = 3 +# Sleeps, in seconds. Tuned live against real applications. They are +# the cost of the method: ~0.12s for a single click. +_DELAY_AFTER_MOVE = 0.015 +_DELAY_AFTER_PRIMER_DOWN = 0.001 +_DELAY_AFTER_PRIMER_UP = 0.100 +_DELAY_AFTER_TARGET_DOWN = 0.001 +_DELAY_BETWEEN_CLICK_PAIRS = 0.080 +# Synthetic focus needs time to be observed by the target, and the final mouse-up +# needs to be consumed BEFORE focus is dropped, or a browser-engine renderer that +# does its own hit-testing misses it. +_DELAY_FOCUS_SETTLE = 0.040 +_DELAY_BEFORE_FOCUS_RELEASE = 0.100 +# ``kCGEventSourceStateHIDSystemState``. Declared HERE rather than imported from +# ``macos_ffi``: that module only ever builds PRIVATE-source events (modifier +# hygiene), so it has no HID constant to borrow, and adding one there would invite +# a public path to use it. See ``_post_recipe`` for why this method needs it. +_K_CG_EVENT_SOURCE_STATE_HID = 1 + +# Field 58's accepted width is unpublished, so the group id is kept in the range +# observed to be accepted (a nanosecond component, always < 1e9). +_CLICK_GROUP_MODULUS = 1_000_000_000 + +#: ``sky_click`` supports single and double clicks only. A triple click is not in +#: the observed recipe, and inventing a third pair would be guessing at ABI. +MAX_SKY_CLICK_COUNT = 2 +MIN_SKY_CLICK_COUNT = 1 + +_REFUSAL_UNAVAILABLE = ( + "click_method 'sky_click' is unavailable on this system ({reason}). Use " + "click_method 'app_post' instead — it delivers to the same application through " + "a public API, and only fails to reach renderers that hit-test against the " + "window server" +) +_REFUSAL_COUNT = ( + "click_method 'sky_click' supports click_count 1 or 2 (got {count}). Use " + "click_method 'app_post' for a triple click" +) +#: ``sky_click`` is a LEFT-button recipe only — see ``types.ERR_SKY_CLICK_BUTTON`` +#: for the wording and ``policy.check_method_button`` for the enforcing gate. The +#: observed private event sequence (the primer pair at (-1,-1), the focus-flag +#: record, the nine private fields) was reverse-engineered for a left click; the +#: button number is one field among those, and there is no evidence the rest of the +#: recipe is button-agnostic. +_REFUSAL_BUTTON = ERR_SKY_CLICK_BUTTON +_REFUSAL_STALE_WINDOW = ( + "the sky_click target window is no longer on screen or no longer belongs to " + "that application. Call computer_get_state again to re-resolve it" +) +_REFUSAL_NO_WINDOW_ID = ( + "click_method 'sky_click' needs a resolved window id, and the driver did not " + "report one for this application. Use click_method 'app_post' instead" +) +_REFUSAL_OUTSIDE_WINDOW = ( + "the sky_click point is outside the target window's bounds. Call " + "computer_get_state again — the window has probably moved or resized" +) +_REFUSAL_PSN = ( + "click_method 'sky_click' could not resolve pid {pid} to a process serial " + "number (OSStatus {status}). Use click_method 'app_post' instead" +) +_REFUSAL_FOCUS = ( + "click_method 'sky_click' could not synthesize target-window focus (OSStatus " + "{status}). Use click_method 'app_post' instead" +) +_REFUSAL_EVENT = "click_method 'sky_click' could not build a mouse event" + +# ONE click at a time, process-wide. The method mutates global window-server focus +# state and restores it afterwards; two interleaved sequences would restore each +# other's state and leave a window synthetically focused. +_dispatch_lock = threading.Lock() +_init_lock = threading.Lock() +_spi: "_SkyLightSPI | None" = None + + +@dataclass(frozen=True) +class SkyLightCapability: + """Whether the private SPI is usable here, and why not when it is not.""" + + missing_symbols: tuple[str, ...] = field(default_factory=tuple) + load_error: str = "" + + @property + def is_available(self) -> bool: + return not self.missing_symbols and not self.load_error + + @property + def reason(self) -> str: + """A model-readable explanation. Empty when available.""" + if self.load_error: + return self.load_error + if self.missing_symbols: + return "missing private symbols: " + ", ".join(self.missing_symbols) + return "" + + +class _SkyLightSPI: + """The bound private symbols, or a capability report explaining the absence. + + Constructed once. Unlike ``macos_ffi``'s bind pass this NEVER raises on a + missing symbol: an unavailable private API is an expected state (a future + macOS, a stripped system), not an error, and the caller turns it into a + refusal that names the public alternative. + """ + + def __init__(self) -> None: + # ``Any``, not ``Callable | None``: these are ctypes function pointers, and + # every caller is reached only after ``capability.is_available`` proved they + # are bound. Narrowing them to Optional would force an ignore at each of the + # five call sites, which reads as noise rather than as a real invariant. + self.post_to_pid: Any = None + self.set_int_field: Any = None + self.set_window_location: Any = None + self.post_event_record: Any = None + self.get_process_for_pid: Any = None + + if not platform_compat.IS_MACOS: + self.capability = SkyLightCapability( + load_error="sky_click requires macOS (the SkyLight framework is Apple-only)" + ) + return + + sky = _dlopen(_SKYLIGHT_PATH) + app_services = _dlopen(_APP_SERVICES_PATH) + if sky is None: + self.capability = SkyLightCapability( + load_error="the private SkyLight framework could not be loaded" + ) + return + + missing: list[str] = [] + self.post_to_pid = _bind( + sky, _SYM_POST_TO_PID, None, [c_int32, c_void_p], missing + ) + self.set_int_field = _bind( + sky, _SYM_SET_INT_FIELD, None, [c_void_p, c_uint32, c_int64], missing + ) + # Scalar doubles, NOT a CGPoint by value: the reference models this private + # ABI as (event, double, double) and notes that relying on the aggregate + # calling convention is what breaks. Keep the scalar form. + self.set_window_location = _bind( + sky, _SYM_SET_WINDOW_LOCATION, None, [c_void_p, c_double, c_double], missing + ) + self.post_event_record = _bind( + sky, _SYM_POST_EVENT_RECORD, c_int32, [c_void_p, POINTER(c_uint8)], missing + ) + if app_services is None: + missing.append(_SYM_GET_PROCESS_FOR_PID) + else: + self.get_process_for_pid = _bind( + app_services, + _SYM_GET_PROCESS_FOR_PID, + c_int32, + [c_int32, c_void_p], + missing, + ) + self.capability = SkyLightCapability(missing_symbols=tuple(missing)) + + +def _dlopen(path: str) -> "ctypes.CDLL | None": + """Load a framework by absolute path, or ``None``. Never raises.""" + try: + return ctypes.CDLL(path) + except OSError: + logger.debug("sky_click: could not load %s", path, exc_info=True) + return None + + +def _bind(lib: Any, symbol: str, restype: Any, argtypes: Any, missing: list) -> Any: + """Bind one symbol with explicit argtypes, recording it when absent. + + ``argtypes`` is mandatory here for the same reason it is in ``macos_ffi``: with + it unset ctypes marshals a Python int as a 32-bit C int and TRUNCATES, which + for a window id or a pid means addressing the wrong window. + """ + try: + fn = getattr(lib, symbol) + except AttributeError: + missing.append(symbol) + return None + fn.restype = restype + fn.argtypes = argtypes + return fn + + +def _get_spi() -> _SkyLightSPI: + global _spi + if _spi is not None: + return _spi + with _init_lock: + if _spi is None: + _spi = _SkyLightSPI() + return _spi + + +def available() -> SkyLightCapability: + """Whether ``sky_click`` can run here. Cheap after the first call. + + Read by the dashboard's config payload and by :func:`sky_click` itself, so the + Settings panel can say "unavailable on this macOS" rather than letting the + operator select a method that always refuses. + """ + return _get_spi().capability + + +def activation_record(window_id: int, *, focused: bool) -> bytes: + """Build the 0xF8-byte window-activation record. + + Pure, and separated from the posting so the byte layout is unit-testable + without a window server. That matters more here than anywhere else in the + package: this is undocumented ABI, so the ONLY way to notice a future edit + silently changing an offset is to pin the bytes. + """ + record = bytearray(_RECORD_SIZE) + record[_RECORD_OFF_SIZE] = _RECORD_SIZE + record[_RECORD_OFF_KIND] = _RECORD_KIND_ACTIVATE + wid = int(window_id) & 0xFFFFFFFF + record[_RECORD_OFF_WINDOW_ID : _RECORD_OFF_WINDOW_ID + 4] = wid.to_bytes(4, "little") + record[_RECORD_OFF_FOCUSED] = _RECORD_FOCUSED_YES if focused else _RECORD_FOCUSED_NO + return bytes(record) + + +@dataclass(frozen=True) +class SkyClickStep: + """One event in the click recipe. See :func:`click_recipe`.""" + + event_type: int + at_target: bool + click_state: int + phase: int + delay_after: float + + +def click_recipe(click_count: int, button: str = _LEFT_BUTTON) -> tuple[SkyClickStep, ...]: + """The ordered event sequence for a ``sky_click``. + + Pure and separately tested, because the ORDER and the primer pair are the + load-bearing parts: a future edit that drops the primer, or reorders the + move-before-down, produces a click on the FRONT window instead of the target — + a silent mis-delivery rather than a failure. + + *button* is accepted only to be REFUSED when it is not the left one. It is a + parameter rather than an unstated assumption because the previous signature took + no button at all and built the recipe with the left-button codes regardless of + what the caller had asked for — so a right-click request became a left click + silently, on a background window the operator cannot see. Naming the argument is + what makes the constraint checkable instead of implicit; see + ``ERR_SKY_CLICK_BUTTON`` for why refusing beats downgrading. + + Raises :class:`ComputerUseError` for an unsupported count OR button; the + dispatcher turns either into a refusal that names ``app_post``. + """ + if not MIN_SKY_CLICK_COUNT <= int(click_count) <= MAX_SKY_CLICK_COUNT: + raise ComputerUseError(_REFUSAL_COUNT.format(count=click_count)) + if button != _LEFT_BUTTON: + raise ComputerUseError(_REFUSAL_BUTTON.format(button=button)) + _, down_type, up_type, _dragged = macos_ffi.mouse_button_codes(_LEFT_BUTTON) + moved_type = macos_ffi.K_CG_EVENT_MOUSE_MOVED + steps = [ + SkyClickStep(moved_type, True, 0, _PHASE_MOVED, _DELAY_AFTER_MOVE), + SkyClickStep(down_type, False, 1, _PHASE_PRIMER_DOWN, _DELAY_AFTER_PRIMER_DOWN), + SkyClickStep(up_type, False, 1, _PHASE_PRIMER_UP, _DELAY_AFTER_PRIMER_UP), + ] + for pair in range(1, int(click_count) + 1): + last = pair == int(click_count) + steps.append( + SkyClickStep(down_type, True, pair, _PHASE_TARGET, _DELAY_AFTER_TARGET_DOWN) + ) + steps.append( + SkyClickStep( + up_type, + True, + pair, + _PHASE_TARGET, + 0.0 if last else _DELAY_BETWEEN_CLICK_PAIRS, + ) + ) + return tuple(steps) + + +def _window_is_current(window_id: int, pid: int) -> bool: + """Is *window_id* still on screen AND still owned by *pid*? + + Re-checked immediately before clicking rather than trusted from the snapshot. + A window id is recycled by the window server, so a stale id can name a + DIFFERENT window — and this method's whole purpose is clicking something the + operator cannot see, where a mis-aimed click is invisible to them. + """ + try: + for info in macos_ffi.window_list(): + if info.window_id == int(window_id): + return info.pid == int(pid) + except Exception: + logger.debug("sky_click: window re-check failed", exc_info=True) + return False + return False + + +def sky_click( + *, + pid: int, + window_id: int, + screen_x: float, + screen_y: float, + window_x: float, + window_y: float, + window_width: float, + window_height: float, + click_count: int = 1, + button: str = _LEFT_BUTTON, +) -> None: + """Click ``(screen_x, screen_y)`` in a BACKGROUND window, without raising it. + + *window_x* / *window_y* are the same point expressed in the window's own + top-left coordinates; the window server routes by the window-local point, so + both are required and the caller (the driver) computes the conversion from the + snapshot's bounds. + + Contracted to raise :class:`ComputerUseError` — never a ctypes error and never + an ``OSError`` — so ``macos_driver``'s ``_guarded`` seam turns every failure + into a model-readable refusal that names ``app_post`` as the alternative. + + **Does not move the operator's pointer** and does not change which app is + frontmost. It DOES briefly mark the target window synthetically focused, and + restores that afterwards; the restore runs even when the click sequence fails. + """ + spi = _get_spi() + capability = spi.capability + if not capability.is_available: + raise ComputerUseError(_REFUSAL_UNAVAILABLE.format(reason=capability.reason)) + if not int(window_id): + raise ComputerUseError(_REFUSAL_NO_WINDOW_ID) + for value in (screen_x, screen_y, window_x, window_y): + if not _finite(value): + raise ComputerUseError(_REFUSAL_OUTSIDE_WINDOW) + if window_width <= 0 or window_height <= 0: + raise ComputerUseError(_REFUSAL_OUTSIDE_WINDOW) + if not (0 <= window_x <= window_width and 0 <= window_y <= window_height): + raise ComputerUseError(_REFUSAL_OUTSIDE_WINDOW) + + recipe = click_recipe(click_count, button) + # Serialized: the synthetic-focus flip is global window-server state. + with _dispatch_lock: + if not _window_is_current(window_id, pid): + raise ComputerUseError(_REFUSAL_STALE_WINDOW) + focused = _begin_synthetic_focus(spi, pid=pid, window_id=window_id) + try: + _post_recipe( + spi, + recipe, + pid=pid, + window_id=window_id, + screen=(screen_x, screen_y), + window=(window_x, window_y), + ) + finally: + if focused: + # SkyLight delivery is asynchronous: drop focus too early and a + # renderer that hit-tests independently never sees the final + # mouse-up, so the click + # registers as a press with no release. + time.sleep(_DELAY_BEFORE_FOCUS_RELEASE) + _end_synthetic_focus(spi, pid=pid, window_id=window_id) + + +def _finite(value: float) -> bool: + return value == value and value not in (float("inf"), float("-inf")) + + +def _post_recipe( + spi: _SkyLightSPI, + recipe: "tuple[SkyClickStep, ...]", + *, + pid: int, + window_id: int, + screen: "tuple[float, float]", + window: "tuple[float, float]", +) -> None: + """Build, stamp and post every step of *recipe*. Releases each event.""" + libs = macos_ffi._frameworks() + # The HID source, NOT the private source every other path in this package + # uses. The window server's routing of these private fields is only observed + # to work with a HID-state source; a private-source event is delivered but + # ignored. The usual modifier-inheritance hazard does not apply because the + # flags are explicitly zeroed on every event below. + source = c_void_p(libs.cg.CGEventSourceCreate(_K_CG_EVENT_SOURCE_STATE_HID)) + if not source: + raise ComputerUseError(_REFUSAL_EVENT) + click_group = int(time.monotonic_ns() % _CLICK_GROUP_MODULUS) + button_number, _down, _up, _dragged = macos_ffi.mouse_button_codes( + _LEFT_BUTTON + ) + try: + for step in recipe: + point = screen if step.at_target else _PRIMER_POINT + local = window if step.at_target else _PRIMER_POINT + event = c_void_p( + libs.cg.CGEventCreateMouseEvent( + source, + c_uint32(int(step.event_type)), + macos_ffi.CGPoint(float(point[0]), float(point[1])), + c_uint32(int(button_number)), + ) + ) + if not event: + raise ComputerUseError(_REFUSAL_EVENT) + try: + libs.cg.CGEventSetFlags(event, ctypes.c_uint64(0)) + _stamp( + spi, + event, + pid=pid, + window_id=window_id, + local=local, + click_state=step.click_state, + phase=step.phase, + click_group=click_group, + ) + # BOTH channels, deliberately, and not as a retry: the SkyLight + # post is what reaches a hit-testing renderer, while the public + # per-pid post preserves plain-AppKit compatibility. Dropping + # either makes one class of app stop responding. + spi.post_to_pid(int(pid), event) + libs.cg.CGEventPostToPid(c_int32(int(pid)), event) + finally: + libs.cf.CFRelease(event) + if step.delay_after: + time.sleep(step.delay_after) + finally: + libs.cf.CFRelease(source) + + +def _stamp( + spi: _SkyLightSPI, + event: c_void_p, + *, + pid: int, + window_id: int, + local: "tuple[float, float]", + click_state: int, + phase: int, + click_group: int, +) -> None: + """Write the private routing fields onto one event.""" + wid = int(window_id) + fields = ( + (_FIELD_GESTURE_PHASE, int(phase)), + (_FIELD_CLICK_STATE, int(click_state)), + (_FIELD_BUTTON_NUMBER, 0), + (_FIELD_SUBTYPE, _EVENT_SUBTYPE_WINDOWED), + (_FIELD_TARGET_PID, int(pid)), + (_FIELD_WINDOW_NUMBER, wid), + (_FIELD_CLICK_GROUP_ID, int(click_group)), + # Both "what is under the pointer" fields name the TARGET window: that is + # the lie that makes the window server route to a background window. + (_FIELD_WINDOW_UNDER_POINTER, wid), + (_FIELD_HANDLING_WINDOW_UNDER_POINTER, wid), + ) + for field_id, value in fields: + spi.set_int_field(event, c_uint32(field_id), c_int64(value)) + spi.set_window_location(event, c_double(local[0]), c_double(local[1])) + + +def _process_serial_number(spi: _SkyLightSPI, pid: int) -> bytes: + """Resolve *pid* to a ProcessSerialNumber, or raise a refusal.""" + buffer = (c_uint8 * _PSN_SIZE)() + status = int(spi.get_process_for_pid(c_int32(int(pid)), buffer)) + if status != 0: + raise ComputerUseError(_REFUSAL_PSN.format(pid=pid, status=status)) + return bytes(buffer) + + +def _post_activation(spi: _SkyLightSPI, psn: bytes, record: bytes) -> None: + psn_buf = (c_uint8 * len(psn)).from_buffer_copy(psn) + rec_buf = (c_uint8 * len(record)).from_buffer_copy(record) + status = int( + spi.post_event_record( + ctypes.cast(psn_buf, c_void_p), + ctypes.cast(rec_buf, POINTER(c_uint8)), + ) + ) + if status != 0: + raise ComputerUseError(_REFUSAL_FOCUS.format(status=status)) + + +def _begin_synthetic_focus(spi: _SkyLightSPI, *, pid: int, window_id: int) -> bool: + """Mark the target window synthetically focused. Returns whether to undo it. + + Skipped when the target is ALREADY frontmost: flipping focus it already has + would be a no-op followed by a deactivate that takes real focus away. + """ + if _is_frontmost(pid): + return False + psn = _process_serial_number(spi, pid) + _post_activation(spi, psn, activation_record(window_id, focused=True)) + time.sleep(_DELAY_FOCUS_SETTLE) + return True + + +def _end_synthetic_focus(spi: _SkyLightSPI, *, pid: int, window_id: int) -> None: + """Undo :func:`_begin_synthetic_focus`. Best-effort; never raises. + + Swallowing here is deliberate: the click already happened, so raising would + replace a successful action with an error. A failed restore leaves the target + marked focused, which the next real click corrects. + """ + try: + psn = _process_serial_number(spi, pid) + _post_activation(spi, psn, activation_record(window_id, focused=False)) + time.sleep(_DELAY_FOCUS_SETTLE) + except Exception: + logger.debug("sky_click: releasing synthetic focus failed", exc_info=True) + + +def _is_frontmost(pid: int) -> bool: + """Is *pid* the owner of the frontmost normal window? + + Read from the CG window list rather than ``NSWorkspace`` (which the Swift + reference uses) because this package deliberately links no AppKit: the front + entry of the on-screen layer-0 list is the same answer without a second + framework. + """ + try: + for info in macos_ffi.window_list(): + if info.layer != macos_ffi.CG_WINDOW_LAYER_NORMAL: + continue + return info.pid == int(pid) + except Exception: + logger.debug("sky_click: frontmost probe failed", exc_info=True) + return False + + +__all__ = [ + "MAX_SKY_CLICK_COUNT", + "MIN_SKY_CLICK_COUNT", + "SkyClickStep", + "SkyLightCapability", + "activation_record", + "available", + "click_recipe", + "sky_click", +] diff --git a/src/kiro_crew/computer_use/overlay.py b/src/kiro_crew/computer_use/overlay.py new file mode 100644 index 00000000000..b7495b57ab9 --- /dev/null +++ b/src/kiro_crew/computer_use/overlay.py @@ -0,0 +1,537 @@ +"""Gateway-side SUPERVISOR for the Cursor Motion overlay process. + +This is the only module the rest of computer use talks to about the fake cursor. +It owns exactly three things: + +1. deciding whether an overlay should exist at all (macOS + the ``cursor_motion`` + opt-in, default OFF); +2. the lifecycle of the ``kiro_crew.computer_use.overlay_proc`` child — lazy + spawn, bounded retry, reap; +3. turning a "the agent is about to click here" event into a + :class:`~kiro_crew.computer_use.cursor_motion.MotionPlan` and shipping it down + the child's stdin. + +**The overlay is PURELY COSMETIC and this module's whole design follows from +that.** It has two absolute properties, and both are asserted by tests: + +* **It never raises into a caller.** Every public method swallows every + exception. A tool call that would have succeeded must not fail because AppKit + was unavailable, the child died, or the pipe was full — a failed overlay + degrades to "no visual cursor", never to a failed tool call. +* **It never blocks the event loop.** The spawn is + ``asyncio.create_subprocess_exec``; the only wait is a bounded + ``asyncio.wait_for`` on the readiness line. The ANIMATION itself is + fire-and-forget: the command is written and the coroutine returns, because the + child draws on its own run loop and making a caller await ~1.4s of decoration + would put a cosmetic subsystem on the latency path of a real action. + +It is also a **no-op by construction** off macOS and when disabled: the enable +check runs before anything else in every method, so a Linux CI shard exercises +these bodies and observes that nothing is spawned. + +Why a separate process at all: AppKit requires a main-thread run loop and the +gateway's main thread IS the asyncio loop. See ``overlay_proc``'s docstring. +""" + +from __future__ import annotations + +import asyncio +import json +import logging +import sys +from typing import Any, Sequence + +from kiro_crew import platform_compat +from kiro_crew.computer_use.cursor_motion import MotionPlan, plan_motion +from kiro_crew.computer_use.types import ( + DEFAULT_CURVE_SCALE, + MAX_CLICK_COUNT, + OVERLAY_CMD_CLICK, + OVERLAY_CMD_HIDE, + OVERLAY_CMD_KEY, + OVERLAY_CMD_MOVE, + OVERLAY_CMD_QUIT, + OVERLAY_KEY_COUNT, + OVERLAY_KEY_MS, + OVERLAY_KEY_POINTS, + OVERLAY_KEY_X, + OVERLAY_KEY_Y, + OVERLAY_MAX_FAILURES, + OVERLAY_MODULE, + OVERLAY_READY_LINE, + OVERLAY_SPAWN_TIMEOUT_SECS, + OVERLAY_STOP_TIMEOUT_SECS, +) + +logger = logging.getLogger(__name__) + +__all__ = [ + "CursorOverlay", + "bind_gateway_loop", + "cursor_motion_enabled", + "get_shared_overlay", + "reset_shared_overlay", + "show_pointer_motion", +] + + +def cursor_motion_enabled() -> bool: + """Whether the desktop cursor overlay may run. + + Two independent conditions, both required: + + * the platform is macOS — the overlay is an AppKit window and there is no + cross-platform equivalent, so every other OS degrades to "no visual cursor" + exactly the way ``UnsupportedBackend`` degrades the driver; + * ``config.json``'s ``computer_use.cursor_motion`` is true — the typed + ``ComputerUseConfig.cursor_motion`` field, default OFF, matching the + reference implementation's opt-in. + + Still read through ``getattr`` despite the field now being declared: this + module is import-safe against a partially-installed build, and a missing + attribute must fall back to OFF rather than raise inside a tool call. + Defaulting to OFF is also the safe direction — an unreadable setting can only + ever mean "no decoration", never "start drawing on the user's screen". + """ + if not platform_compat.IS_MACOS: + return False + try: + from kiro_crew.config.loader import KiroCrewConfig + + section = getattr(KiroCrewConfig.load(), "computer_use", None) + except Exception: + logger.debug("cursor-motion config unavailable; overlay disabled", exc_info=True) + return False + value = getattr(section, "cursor_motion", False) + return value is True + + +class CursorOverlay: + """Supervises at most one overlay child process. + + Serialized by a single :class:`asyncio.Lock`: the child's stdin is an ordered + byte stream and two concurrent writers would interleave half-lines, so every + public method takes the lock for the duration of its spawn-and-write. The lock + is created lazily because this object is constructed at import-adjacent time + (the shared singleton) and an ``asyncio.Lock`` built outside a running loop is + the cross-loop hazard ``kiro_crew.__init__`` documents at length. + """ + + def __init__(self) -> None: + self._proc: "asyncio.subprocess.Process | None" = None + self._lock: "asyncio.Lock | None" = None + self._failures = 0 + # Last tip position, so a move starts where the cursor actually is rather + # than teleporting from a fixed corner. ``None`` means "never drawn". + self._last_point: "tuple[float, float] | None" = None + + # ── public surface ── + + async def move_to( + self, + x: float, + y: float, + *, + curve_scale: float = DEFAULT_CURVE_SCALE, + ) -> bool: + """Animate the fake cursor to the TOP-LEFT screen point ``(x, y)``. + + Returns whether a command was actually shipped — useful to a test and to + the SEL audit trail, and ignored by every real caller, because the answer + "no" is a perfectly acceptable outcome for a decoration. + """ + if not cursor_motion_enabled(): + return False + try: + start = self._last_point or (float(x), float(y)) + plan = plan_motion(start, (float(x), float(y)), curve_scale=curve_scale) + sent = await self._send(_move_command(plan)) + if sent: + self._last_point = (float(x), float(y)) + return sent + except Exception: + # Belt and braces: ``_send`` already swallows, so reaching here means a + # bug in the planner. Still not allowed to reach the caller. + logger.debug("cursor-motion move failed", exc_info=True) + return False + + async def pulse_click(self, x: float, y: float, count: int = 1) -> bool: + """Draw *count* click pulses at ``(x, y)`` (top-left screen coordinates).""" + if not cursor_motion_enabled(): + return False + try: + pulses = min(max(int(count), 1), MAX_CLICK_COUNT) + command = { + OVERLAY_CMD_KEY: OVERLAY_CMD_CLICK, + OVERLAY_KEY_X: float(x), + OVERLAY_KEY_Y: float(y), + OVERLAY_KEY_COUNT: pulses, + } + sent = await self._send(command) + if sent: + self._last_point = (float(x), float(y)) + return sent + except Exception: + logger.debug("cursor-motion click failed", exc_info=True) + return False + + async def hide(self) -> bool: + """Order the fake cursor off screen, keeping the child alive for reuse. + + Does NOT spawn: hiding a cursor that was never drawn is a no-op, and + starting a process in order to hide nothing would be absurd. + """ + if self._proc is None: + return False + try: + return await self._send({OVERLAY_CMD_KEY: OVERLAY_CMD_HIDE}, spawn=False) + except Exception: + logger.debug("cursor-motion hide failed", exc_info=True) + return False + + async def stop(self) -> None: + """Tear the child down: ``quit``, close stdin, reap, then force-kill. + + Three escalating steps, all bounded: + + 1. a ``quit`` command, which lets the child order its window out cleanly; + 2. closing stdin — EOF is the child's primary exit path and the one that + also covers a gateway crash, where step 1 never happened; + 3. after :data:`OVERLAY_STOP_TIMEOUT_SECS`, ``kill_process_tree`` via + ``platform_compat`` (never a raw ``os.killpg``). + + Idempotent and never raises, so it is safe from a shutdown handler. + """ + async with self._get_lock(): + proc = self._proc + self._proc = None + self._last_point = None + if proc is None: + return + try: + await self._write_line(proc, {OVERLAY_CMD_KEY: OVERLAY_CMD_QUIT}) + except Exception: + logger.debug("cursor-motion quit write failed", exc_info=True) + try: + if proc.stdin is not None and not proc.stdin.is_closing(): + proc.stdin.close() + except Exception: + logger.debug("cursor-motion stdin close failed", exc_info=True) + try: + await asyncio.wait_for(proc.wait(), timeout=OVERLAY_STOP_TIMEOUT_SECS) + return + # BOTH exception names, deliberately: on Python 3.11+ they are the same + # class, but on 3.10 (which CI gates on) ``asyncio.TimeoutError`` does NOT + # inherit from the builtin. Catching only one would let a timeout fall into + # the ``except Exception`` below, which RETURNS — skipping the kill and + # leaving the overlay child alive with a fake cursor on the user's screen. + # Same reasoning at the other two wait_for sites in this module. + except (asyncio.TimeoutError, TimeoutError): + logger.debug("cursor-motion child ignored EOF; killing") + except Exception: + logger.debug("cursor-motion child wait failed", exc_info=True) + return + # Route the kill through platform_compat: a raw ``os.killpg`` is a POSIX-only + # call and ``os.kill(pid, 0)`` TERMINATES on Windows. This code path cannot + # run off macOS today, but the shim is the repo-wide contract. + try: + platform_compat.kill_process_tree(proc.pid, platform_compat.SIGKILL) + except Exception: + logger.debug("cursor-motion kill failed", exc_info=True) + try: + await asyncio.wait_for(proc.wait(), timeout=OVERLAY_STOP_TIMEOUT_SECS) + except Exception: + # A zombie we cannot reap is still better than raising from shutdown. + logger.debug("cursor-motion child did not exit after kill", exc_info=True) + + @property + def running(self) -> bool: + """Whether a live child process is currently supervised.""" + proc = self._proc + return proc is not None and proc.returncode is None + + # ── internals ── + + def _get_lock(self) -> asyncio.Lock: + """The write lock, created inside the running loop on first use.""" + if self._lock is None: + self._lock = asyncio.Lock() + return self._lock + + async def _send(self, command: "dict[str, Any]", *, spawn: bool = True) -> bool: + """Ship one command, spawning the child first if needed. + + Returns False on every failure and logs at debug. The failure counter is + what stops a broken AppKit from becoming a respawn loop: after + :data:`OVERLAY_MAX_FAILURES` consecutive failures the supervisor gives up + for the life of the process, because a cursor that cannot be drawn is + cosmetic but a spawn loop is a real resource leak. + """ + if self._failures >= OVERLAY_MAX_FAILURES: + return False + async with self._get_lock(): + proc = self._proc + if proc is not None and proc.returncode is not None: + # The child exited on its own (crash, or a manual kill). Forget it + # so the next command spawns a fresh one instead of writing into a + # dead pipe. + logger.debug("cursor-motion child exited (rc=%s)", proc.returncode) + proc = None + self._proc = None + self._last_point = None + if proc is None: + if not spawn: + return False + proc = await self._spawn() + if proc is None: + return False + ok = await self._write_line(proc, command) + if ok: + self._failures = 0 + else: + self._failures += 1 + # A failed write means the pipe is unusable; drop the child so the + # next attempt starts clean rather than writing into it again. + self._proc = None + await self._reap(proc) + return ok + + async def _spawn(self) -> "asyncio.subprocess.Process | None": + """Start the overlay child and wait (bounded) for its readiness line. + + The argv is fixed: `` -m + kiro_crew.computer_use.overlay_proc``. Nothing agent-supplied enters it — + the only agent-influenced values in this whole subsystem are the numeric + coordinates, and those travel as JSON on stdin, never as argv. + + ``start_new_session`` / ``creationflags`` are passed explicitly per the + repo's spawn-isolation contract, so the child sits in its own process group + and :func:`platform_compat.kill_process_tree` can reap it. + """ + argv = [sys.executable, "-m", OVERLAY_MODULE] + try: + proc = await asyncio.create_subprocess_exec( + *argv, + stdin=asyncio.subprocess.PIPE, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.DEVNULL, + start_new_session=platform_compat.IS_POSIX, + creationflags=platform_compat.CREATE_NEW_PROCESS_GROUP, + ) + except Exception: + logger.debug("cursor-motion spawn failed", exc_info=True) + self._failures += 1 + return None + if not await self._await_ready(proc): + self._failures += 1 + await self._reap(proc) + return None + self._proc = proc + return proc + + async def _await_ready(self, proc: "asyncio.subprocess.Process") -> bool: + """Wait (bounded) for the child's ``KIROCREW_OVERLAY_READY`` line. + + Bounded because the alternative is a coroutine that hangs forever on a + child that wedged before it printed anything — and this coroutine is + awaited from a tool-call path, so an unbounded wait here would stall a real + action for a decoration. + + A ``ready 0`` line (the child started but could not build a window) is + treated as a failure so the supervisor's give-up counter advances rather + than the gateway shipping commands into a process that will never draw. + """ + if proc.stdout is None: + return False + try: + line = await asyncio.wait_for( + proc.stdout.readline(), timeout=OVERLAY_SPAWN_TIMEOUT_SECS + ) + except (asyncio.TimeoutError, TimeoutError): + logger.debug("cursor-motion child never reported ready") + return False + except Exception: + logger.debug("cursor-motion readiness read failed", exc_info=True) + return False + text = line.decode("utf-8", errors="replace").strip() if line else "" + if not text.startswith(OVERLAY_READY_LINE): + logger.debug("cursor-motion child said %r instead of ready", text[:80]) + return False + return text.split()[-1] != "0" + + async def _write_line( + self, proc: "asyncio.subprocess.Process", command: "dict[str, Any]" + ) -> bool: + """Write one NDJSON command and drain. Never raises.""" + stdin = proc.stdin + if stdin is None or stdin.is_closing(): + return False + try: + payload = json.dumps(command, separators=(",", ":"), allow_nan=False) + except (TypeError, ValueError): + logger.debug("cursor-motion command not serializable", exc_info=True) + return False + try: + stdin.write(f"{payload}\n".encode("utf-8")) + await stdin.drain() + return True + except Exception: + # BrokenPipeError / ConnectionResetError when the child is gone. + logger.debug("cursor-motion write failed", exc_info=True) + return False + + async def _reap(self, proc: "asyncio.subprocess.Process") -> None: + """Best-effort teardown of a child we are abandoning. + + Always runs to completion so an abandoned child cannot survive as an orphan + with a window on the user's screen: close stdin (its EOF exit), wait, and + force-kill through ``platform_compat`` if it ignores both. + """ + try: + if proc.stdin is not None and not proc.stdin.is_closing(): + proc.stdin.close() + except Exception: + logger.debug("cursor-motion reap stdin close failed", exc_info=True) + try: + await asyncio.wait_for(proc.wait(), timeout=OVERLAY_STOP_TIMEOUT_SECS) + return + except (asyncio.TimeoutError, TimeoutError): + pass + except Exception: + logger.debug("cursor-motion reap wait failed", exc_info=True) + return + try: + platform_compat.kill_process_tree(proc.pid, platform_compat.SIGKILL) + except Exception: + logger.debug("cursor-motion reap kill failed", exc_info=True) + try: + await asyncio.wait_for(proc.wait(), timeout=OVERLAY_STOP_TIMEOUT_SECS) + except Exception: + logger.debug("cursor-motion reap did not complete", exc_info=True) + + +def _move_command(plan: MotionPlan) -> "dict[str, Any]": + """Serialize a :class:`MotionPlan` into the ``move`` wire command. + + Points go as ``[[x, y], ...]`` pairs rather than ``{"x":..,"y":..}`` objects: + a 96-point path is the largest thing this protocol ever carries, and the pair + form is roughly a third of the bytes for the same information. + """ + return { + OVERLAY_CMD_KEY: OVERLAY_CMD_MOVE, + OVERLAY_KEY_POINTS: [[point[0], point[1]] for point in plan.points], + OVERLAY_KEY_MS: plan.duration_ms, + } + + +def points_payload(points: Sequence[tuple[float, float]]) -> "list[list[float]]": + """Wire form for an arbitrary point sequence (used by tests and diagnostics).""" + return [[float(x), float(y)] for x, y in points] + + +# ── Process-wide shared supervisor ── +# One overlay per process, for the same reason there is one backend and one +# snapshot cache: a second supervisor would spawn a second child and two fake +# cursors would fight over the same screen. + +_shared_overlay: "CursorOverlay | None" = None + + +def get_shared_overlay() -> CursorOverlay: + """Process-wide :class:`CursorOverlay` singleton. + + No lock: construction is a handful of attribute assignments with no I/O, and + the gateway's callers all live on the one event loop. The ``asyncio.Lock`` + that actually matters is created inside the instance, in the running loop. + """ + global _shared_overlay + if _shared_overlay is None: + _shared_overlay = CursorOverlay() + return _shared_overlay + + +def reset_shared_overlay() -> None: + """Drop the shared supervisor WITHOUT reaping its child (tests only). + + Deliberately does not stop the process: this is sync, ``stop`` is async, and a + sync function that spawned a task to kill a process would be a worse hazard + than the leak it avoided. Production shutdown calls ``await stop()``; tests + that spawned a real child must do the same before resetting. + """ + global _shared_overlay + _shared_overlay = None + + +def show_pointer_motion(x: float, y: float, count: int = 1) -> None: + """Animate the visible cursor to ``(x, y)`` and pulse it. **Sync, fire-and-forget.** + + This is the seam the BLOCKING dispatcher calls (``tools._perform``, on a + worker thread) immediately before a real-pointer click or drag. Three + properties make it safe from there: + + * **it never blocks the caller.** The animation is scheduled onto the gateway's + event loop with ``run_coroutine_threadsafe`` and the future is NOT awaited. + Waiting for the glide would add its duration to every pointer click's latency + for a purely cosmetic effect, and a wedged AppKit child would then stall the + tool call itself; + * **it never raises.** A decoration must not be able to turn a successful click + into a failed tool call, so every failure — no running loop, a dead child, a + full pipe — is swallowed at debug; + * **it is not a permit.** By the time this runs the click has already been + authorized upstream and its method resolved to a pointer-moving one. Drawing + a cursor grants nothing, and skipping the drawing denies nothing. + + Ordering is best-effort by design: the click may land a few milliseconds before + the drawn cursor finishes its glide. Making it strictly-before would mean + awaiting an animation on the critical path, which is the trade this rejects. + """ + if not cursor_motion_enabled(): + return + try: + loop = _gateway_loop() + if loop is None: + return + overlay = get_shared_overlay() + + async def _animate() -> None: + await overlay.move_to(x, y) + await overlay.pulse_click(x, y, count) + + asyncio.run_coroutine_threadsafe(_animate(), loop) + except Exception: + logger.debug("cursor-motion pre-click animation could not be scheduled", exc_info=True) + + +def _gateway_loop() -> "asyncio.AbstractEventLoop | None": + """The gateway's event loop, or ``None`` when there is not one. + + Recorded by :func:`bind_gateway_loop` at gateway start rather than discovered: + this is called from a worker thread, where ``get_running_loop`` raises and + ``get_event_loop`` would either create a fresh unrun loop or fail depending on + the Python version — and a coroutine scheduled onto a loop nobody runs would + simply never execute. + """ + loop = _bound_loop + if loop is None or loop.is_closed(): + return None + return loop + + +_bound_loop: "asyncio.AbstractEventLoop | None" = None + + +def bind_gateway_loop(loop: "asyncio.AbstractEventLoop | None" = None) -> None: + """Record the loop that :func:`show_pointer_motion` should schedule onto. + + Called from the async invoke handler (which runs ON that loop) rather than from + a startup hook, so the binding cannot go stale across a gateway restart and no + lifecycle wiring is needed for a feature that is off by default. + """ + global _bound_loop + if loop is not None: + _bound_loop = loop + return + try: + _bound_loop = asyncio.get_running_loop() + except RuntimeError: + _bound_loop = None diff --git a/src/kiro_crew/computer_use/overlay_proc.py b/src/kiro_crew/computer_use/overlay_proc.py new file mode 100644 index 00000000000..3f39827e719 --- /dev/null +++ b/src/kiro_crew/computer_use/overlay_proc.py @@ -0,0 +1,768 @@ +"""The Cursor Motion overlay RENDERER — a separate process, run as ``python -m``. + + python -m kiro_crew.computer_use.overlay_proc + +Reads newline-delimited JSON commands on stdin and draws a fake mouse cursor on +the real desktop. That is its entire job: it makes no policy decisions, reads no +config, computes no paths, and never touches the accessibility or capture +surfaces. Every "should we?" question was already answered by +:mod:`kiro_crew.computer_use.overlay` before a byte reached this process. + +**Why this module owns its own ctypes, when ``macos_ffi`` is otherwise the only +module allowed to.** That invariant exists so the FFI hazards of the AX/CG +surface — the segfault-on-missing-argtypes, the CFString lifetime discipline, the +uncatchable ``CFArrayGetValueAtIndex`` abort — are audited in ONE file inside the +gateway process. This module is not in the gateway process. It is a separate +executable whose entire address space is disposable: it touches the AppKit/ObjC +runtime (a surface ``macos_ffi`` does not model at all), it needs a main-thread +run loop that the gateway's main thread cannot provide because that thread is the +asyncio loop, and if it segfaults the only consequence is that no fake cursor is +drawn. Merging this into ``macos_ffi`` would drag AppKit into the gateway's +address space to no benefit and would put a run-loop pump next to code that must +never block. The separation is the safety property, not a violation of it. + +Hard-won FFI facts encoded below, each of which cost a real debugging cycle: + +* **``objc_msgSend`` is a SINGLE ctypes function object.** Assigning + ``restype``/``argtypes`` mutates it GLOBALLY, so a cached, pre-configured + binding goes stale the instant anything else calls it — the observed symptom was + ``TypeError: this function takes at least 4 arguments``. :func:`_msg` therefore + re-declares the signature at every call site. It looks wasteful; it is the only + correct pattern short of hand-rolling separate function pointers. +* **Every symbol needs BOTH ``restype`` and ``argtypes``.** A missing ``argtypes`` + makes ctypes marshal a Python int as a 32-bit C int and TRUNCATE a 64-bit + pointer, which is a SIGSEGV rather than an exception. There is no partial + declaration in this file. +* **``setSharingType: 0`` (NSWindowSharingNone) makes the overlay invisible to + ``screencapture``** — A/B verified. It stays on: the agent's own decoration must + never pollute the screenshots the agent takes, which would otherwise feed a fake + cursor back into the model's observations as if it were part of the UI. +* **NSWindow's origin is BOTTOM-left; the rest of computer use is TOP-left.** The + single flip lives in :func:`_place`, so no other module has to think about it. +* **``setIgnoresMouseEvents:True``** makes the window click-THROUGH. Without it a + purely decorative window would swallow the user's own clicks — turning a + cosmetic feature into an input-blocking bug. + +Lifecycle: the process exits cleanly on stdin EOF. That is the primary guarantee +that a crashed gateway cannot leave an orphan cursor parked on the user's screen — +when the parent dies its end of the pipe closes, ``readline`` returns ``""``, and +this process orders the window out and returns. :data:`OVERLAY_IDLE_HIDE_SECS` is +the backstop for the rarer case where the pipe stays open but nobody writes. +""" + +from __future__ import annotations + +import ctypes +import ctypes.util +import json +import logging +import math +import os +import sys +import threading +import time +from typing import Any, Sequence + +from kiro_crew import platform_compat +from kiro_crew.computer_use.types import ( + CLICK_PULSE_DEPTH, + CLICK_PULSE_GAP_MS, + CLICK_PULSE_MS, + CURSOR_GLYPH_HEIGHT, + CURSOR_GLYPH_WIDTH, + CURSOR_HOTSPOT_X, + CURSOR_HOTSPOT_Y, + FALLBACK_SCREEN_HEIGHT, + FALLBACK_SCREEN_WIDTH, + MAX_CLICK_COUNT, + MAX_MOVE_DURATION_MS, + MIN_MOVE_DURATION_MS, + NS_ACTIVATION_POLICY_ACCESSORY, + NS_BACKING_STORE_BUFFERED, + NS_COLLECTION_BEHAVIOR, + NS_IMAGE_SCALE_PROPORTIONAL, + NS_STATUS_WINDOW_LEVEL, + NS_WINDOW_SHARING_NONE, + NS_WINDOW_STYLE_BORDERLESS, + OVERLAY_CMD_CLICK, + OVERLAY_CMD_HIDE, + OVERLAY_CMD_KEY, + OVERLAY_CMD_MOVE, + OVERLAY_CMD_QUIT, + OVERLAY_FRAME_SLICE_SECS, + OVERLAY_IDLE_HIDE_SECS, + OVERLAY_IDLE_SLICE_SECS, + OVERLAY_KEY_COUNT, + OVERLAY_KEY_MS, + OVERLAY_KEY_POINTS, + OVERLAY_KEY_X, + OVERLAY_KEY_Y, + OVERLAY_READY_LINE, +) + +logger = logging.getLogger(__name__) + +__all__ = ["CursorOverlayWindow", "ObjCRuntime", "main", "read_commands"] + +# ── ctypes struct layouts ── +# Real ``ctypes.Structure`` types, never "two doubles": a CGPoint passed as two +# separate arguments is mis-marshalled on arm64 (the same class of bug that made +# ``CGEventCreateMouseEvent`` misbehave in the FFI probe). + + +class CGPoint(ctypes.Structure): + """CoreGraphics point. Bottom-left origin when it reaches an NSWindow.""" + + _fields_ = [("x", ctypes.c_double), ("y", ctypes.c_double)] + + +class CGSize(ctypes.Structure): + """CoreGraphics size.""" + + _fields_ = [("width", ctypes.c_double), ("height", ctypes.c_double)] + + +class CGRect(ctypes.Structure): + """CoreGraphics rect (origin + size).""" + + _fields_ = [("origin", CGPoint), ("size", CGSize)] + + +_RUNLOOP_MODE = b"kCFRunLoopDefaultMode" + + +class ObjCRuntime: + """A minimal, self-contained binding to the ObjC runtime and AppKit. + + Instantiating this is what LOADS the native libraries, so nothing happens at + import time and this module can be imported (and its pure helpers tested) on a + Linux CI shard. Tests substitute a fake with the same three primitives — + ``cls`` / ``sel`` / ``msg`` — which is the whole reason the window code below + talks to this object rather than to ``ctypes`` directly. + """ + + def __init__(self) -> None: + objc_path = ctypes.util.find_library("objc") + if not objc_path: # pragma: no cover - present on every macOS + raise OSError("libobjc not found") + self._objc = ctypes.CDLL(objc_path) + appkit_path = ctypes.util.find_library("AppKit") + if appkit_path: + # Loading AppKit REALIZES the NSApplication/NSWindow classes so + # ``objc_getClass`` can find them. The handle itself is unused. + ctypes.CDLL(appkit_path) + cg_path = ctypes.util.find_library("CoreGraphics") + self._cg = ctypes.CDLL(cg_path) if cg_path else None + + # BOTH restype and argtypes on every symbol — see the module docstring. + self._objc.objc_getClass.restype = ctypes.c_void_p + self._objc.objc_getClass.argtypes = [ctypes.c_char_p] + self._objc.sel_registerName.restype = ctypes.c_void_p + self._objc.sel_registerName.argtypes = [ctypes.c_char_p] + if self._cg is not None: + self._cg.CGMainDisplayID.restype = ctypes.c_uint32 + self._cg.CGMainDisplayID.argtypes = [] + self._cg.CGDisplayPixelsWide.restype = ctypes.c_size_t + self._cg.CGDisplayPixelsWide.argtypes = [ctypes.c_uint32] + self._cg.CGDisplayPixelsHigh.restype = ctypes.c_size_t + self._cg.CGDisplayPixelsHigh.argtypes = [ctypes.c_uint32] + + def cls(self, name: str) -> Any: + """``objc_getClass`` — the class object for *name*, or ``None``.""" + return self._objc.objc_getClass(name.encode("utf-8")) + + def sel(self, name: str) -> Any: + """``sel_registerName`` — the selector for *name*.""" + return self._objc.sel_registerName(name.encode("utf-8")) + + def msg( + self, receiver: Any, selector: Any, restype: Any, argtypes: Sequence[Any], *args: Any + ) -> Any: + """Send *selector* to *receiver*, declaring the signature AT THE CALL. + + ``objc_msgSend`` is one shared ctypes function object, so its + ``restype``/``argtypes`` are process-global mutable state: caching a + configured binding and reusing it later is the bug that raised + ``TypeError: this function takes at least 4 arguments`` in the prototype. + Re-declaring on every send is the fix. + """ + fn = self._objc.objc_msgSend + fn.restype = restype + fn.argtypes = [ctypes.c_void_p, ctypes.c_void_p] + list(argtypes) + return fn(receiver, selector, *args) + + def screen_size(self) -> tuple[float, float]: + """Main-display size in points, or the fallback constants. + + Only used to clamp a target point into something finite. An approximate + clamp on an exotic multi-display setup is strictly better than refusing to + draw, because the overlay is cosmetic. + """ + if self._cg is None: # pragma: no cover - CoreGraphics is always present + return (FALLBACK_SCREEN_WIDTH, FALLBACK_SCREEN_HEIGHT) + try: + display = self._cg.CGMainDisplayID() + width = float(self._cg.CGDisplayPixelsWide(display)) + height = float(self._cg.CGDisplayPixelsHigh(display)) + except Exception: + logger.debug("overlay: display size probe failed", exc_info=True) + return (FALLBACK_SCREEN_WIDTH, FALLBACK_SCREEN_HEIGHT) + if width <= 0.0 or height <= 0.0: + return (FALLBACK_SCREEN_WIDTH, FALLBACK_SCREEN_HEIGHT) + return (width, height) + + +class CursorOverlayWindow: + """The borderless, click-through, screenshot-invisible NSWindow. + + Constructed lazily by :meth:`ensure` so a process that receives only a + ``quit`` never creates a window at all. Every method is best-effort: a failure + to draw is logged at debug and swallowed, because this process exists purely to + put pixels on a screen and the caller (the gateway) has already committed to + treating a missing cursor as a non-event. + """ + + def __init__(self, runtime: ObjCRuntime) -> None: + self._rt = runtime + self._window: Any = None + self._glyph_width = CURSOR_GLYPH_WIDTH + self._glyph_height = CURSOR_GLYPH_HEIGHT + self._hotspot_x = CURSOR_HOTSPOT_X + self._hotspot_y = CURSOR_HOTSPOT_Y + self._screen_width, self._screen_height = runtime.screen_size() + self._visible = False + + # ── construction ── + + def ensure(self) -> bool: + """Create the window if needed. Returns whether one exists.""" + if self._window is not None: + return True + try: + self._build() + except Exception: + logger.debug("overlay: window construction failed", exc_info=True) + self._window = None + return False + return self._window is not None + + def _build(self) -> None: + """Create NSApplication + the overlay NSWindow and add the glyph view.""" + rt = self._rt + app = rt.msg(rt.cls("NSApplication"), rt.sel("sharedApplication"), ctypes.c_void_p, []) + # Accessory activation policy: no Dock icon, no menu bar, and — critically + # — the overlay never becomes the active application, so it cannot steal + # focus from the app the agent is driving. + rt.msg( + app, + rt.sel("setActivationPolicy:"), + ctypes.c_bool, + [ctypes.c_long], + NS_ACTIVATION_POLICY_ACCESSORY, + ) + + image, hotspot = self._load_arrow_glyph() + + window = rt.msg(rt.cls("NSWindow"), rt.sel("alloc"), ctypes.c_void_p, []) + window = rt.msg( + window, + rt.sel("initWithContentRect:styleMask:backing:defer:"), + ctypes.c_void_p, + [CGRect, ctypes.c_ulong, ctypes.c_ulong, ctypes.c_bool], + CGRect(CGPoint(0.0, 0.0), CGSize(self._glyph_width, self._glyph_height)), + NS_WINDOW_STYLE_BORDERLESS, + NS_BACKING_STORE_BUFFERED, + False, + ) + if not window: + raise OSError("NSWindow allocation returned NULL") + + clear = rt.msg(rt.cls("NSColor"), rt.sel("clearColor"), ctypes.c_void_p, []) + rt.msg(window, rt.sel("setBackgroundColor:"), None, [ctypes.c_void_p], clear) + rt.msg(window, rt.sel("setOpaque:"), None, [ctypes.c_bool], False) + rt.msg(window, rt.sel("setHasShadow:"), None, [ctypes.c_bool], False) + # CLICK-THROUGH. Without this a decorative window would swallow the user's + # own clicks in the region it covers. + rt.msg(window, rt.sel("setIgnoresMouseEvents:"), None, [ctypes.c_bool], True) + rt.msg(window, rt.sel("setLevel:"), None, [ctypes.c_long], NS_STATUS_WINDOW_LEVEL) + # Invisible to screencapture / CGWindowList — keeps the agent's own fake + # cursor out of the screenshots the agent takes. Do not relax this. + rt.msg(window, rt.sel("setSharingType:"), None, [ctypes.c_long], NS_WINDOW_SHARING_NONE) + rt.msg( + window, + rt.sel("setCollectionBehavior:"), + None, + [ctypes.c_ulong], + NS_COLLECTION_BEHAVIOR, + ) + rt.msg(window, rt.sel("setAlphaValue:"), None, [ctypes.c_double], 1.0) + + if image: + view = rt.msg(rt.cls("NSImageView"), rt.sel("alloc"), ctypes.c_void_p, []) + view = rt.msg( + view, + rt.sel("initWithFrame:"), + ctypes.c_void_p, + [CGRect], + CGRect(CGPoint(0.0, 0.0), CGSize(self._glyph_width, self._glyph_height)), + ) + rt.msg(view, rt.sel("setImage:"), None, [ctypes.c_void_p], image) + rt.msg( + view, + rt.sel("setImageScaling:"), + None, + [ctypes.c_ulong], + NS_IMAGE_SCALE_PROPORTIONAL, + ) + content = rt.msg(window, rt.sel("contentView"), ctypes.c_void_p, []) + if content: + rt.msg(content, rt.sel("addSubview:"), None, [ctypes.c_void_p], view) + + self._hotspot_x, self._hotspot_y = hotspot + self._window = window + + def _load_arrow_glyph(self) -> tuple[Any, tuple[float, float]]: + """The system arrow cursor's image, its size, and its hot spot. + + Using ``NSCursor arrowCursor`` rather than shipping artwork means the fake + cursor matches the user's real one (including accessibility size settings), + which is the difference between "the agent is pointing at this" and "there + is a weird glyph on my screen". Measured 28x40 with a (5,5) hot spot on the + probe machine; the constants in ``types`` are only the fallback. + + The hot spot is returned in TOP-LEFT glyph coordinates (AppKit reports it + that way), and :meth:`_place` converts once. + """ + rt = self._rt + try: + cursor = rt.msg(rt.cls("NSCursor"), rt.sel("arrowCursor"), ctypes.c_void_p, []) + if not cursor: + return (None, (self._hotspot_x, self._hotspot_y)) + image = rt.msg(cursor, rt.sel("image"), ctypes.c_void_p, []) + if not image: + return (None, (self._hotspot_x, self._hotspot_y)) + size = rt.msg(image, rt.sel("size"), CGSize, []) + width = float(getattr(size, "width", 0.0) or 0.0) + height = float(getattr(size, "height", 0.0) or 0.0) + if width > 0.0 and height > 0.0: + self._glyph_width, self._glyph_height = width, height + spot = rt.msg(cursor, rt.sel("hotSpot"), CGPoint, []) + hx = float(getattr(spot, "x", CURSOR_HOTSPOT_X) or 0.0) + hy = float(getattr(spot, "y", CURSOR_HOTSPOT_Y) or 0.0) + return (image, (hx, hy)) + except Exception: + logger.debug("overlay: arrow glyph probe failed", exc_info=True) + return (None, (self._hotspot_x, self._hotspot_y)) + + # ── drawing ── + + def show(self) -> None: + """Order the window in front without activating this application. + + ``orderFrontRegardless`` (not ``makeKeyAndOrderFront:``) is deliberate: the + latter would activate the overlay process and pull focus away from the + application the agent is driving, which would change that app's behaviour — + a cosmetic feature must not do that. + """ + if not self.ensure(): + return + try: + self._rt.msg(self._window, self._rt.sel("orderFrontRegardless"), None, []) + self._rt.msg(self._window, self._rt.sel("setAlphaValue:"), None, [ctypes.c_double], 1.0) + self._visible = True + except Exception: + logger.debug("overlay: show failed", exc_info=True) + + def hide(self) -> None: + """Order the window out. Safe to call when nothing was ever created.""" + if self._window is None: + self._visible = False + return + try: + self._rt.msg(self._window, self._rt.sel("orderOut:"), None, [ctypes.c_void_p], None) + except Exception: + logger.debug("overlay: hide failed", exc_info=True) + self._visible = False + + def close(self) -> None: + """Hide and release the window — the teardown path on EOF/quit.""" + self.hide() + if self._window is None: + return + try: + self._rt.msg(self._window, self._rt.sel("close"), None, []) + except Exception: + logger.debug("overlay: close failed", exc_info=True) + self._window = None + + def move_along(self, points: Sequence[tuple[float, float]], duration_ms: int) -> None: + """Animate the tip along *points* over *duration_ms*, pumping the run loop. + + Frame pacing is by WALL CLOCK against the requested duration, not by + "one point per pumped frame": the point list is a shape, not a schedule, so + a slow frame skips ahead rather than stretching the animation. The tip is + always placed at the LAST point before returning, so a skipped frame can + never leave the cursor short of the target the caller asked for. + """ + if not points: + return + if not self.ensure(): + return + self.show() + duration = min(max(int(duration_ms), MIN_MOVE_DURATION_MS), MAX_MOVE_DURATION_MS) / 1000.0 + started = time.monotonic() + last = len(points) - 1 + while True: + elapsed = time.monotonic() - started + progress = 1.0 if duration <= 0.0 else min(max(elapsed / duration, 0.0), 1.0) + index = min(int(round(progress * last)), last) + self._place(points[index][0], points[index][1]) + if progress >= 1.0: + break + self.pump(OVERLAY_FRAME_SLICE_SECS) + self._place(points[last][0], points[last][1]) + self.pump(OVERLAY_FRAME_SLICE_SECS) + + def pulse_click(self, x: float, y: float, count: int) -> None: + """Draw *count* click pulses at the given TOP-LEFT point. + + One sine half-period of alpha dip per click. Alpha (rather than a scale + transform) because scaling the window would move its content rect and + therefore its tip anchor, so the "click" would visibly drift off the + element it is announcing. + """ + if not self.ensure(): + return + self.show() + self._place(x, y) + pulses = min(max(int(count), 1), MAX_CLICK_COUNT) + duration = CLICK_PULSE_MS / 1000.0 + for pulse in range(pulses): + started = time.monotonic() + while True: + elapsed = time.monotonic() - started + progress = 1.0 if duration <= 0.0 else min(max(elapsed / duration, 0.0), 1.0) + self._set_alpha(1.0 - CLICK_PULSE_DEPTH * math.sin(progress * math.pi)) + if progress >= 1.0: + break + self.pump(OVERLAY_FRAME_SLICE_SECS) + self._set_alpha(1.0) + if pulse < pulses - 1: + self.pump(CLICK_PULSE_GAP_MS / 1000.0) + + def pump(self, seconds: float) -> None: + """Run the AppKit run loop for *seconds*. + + Manual pumping rather than ``NSApp run``: that call never returns, and this + process must stay in control of its own stdin-reading loop so the EOF exit + (the anti-orphan-window guarantee) keeps working. + """ + rt = self._rt + try: + run_loop = rt.msg(rt.cls("NSRunLoop"), rt.sel("currentRunLoop"), ctypes.c_void_p, []) + date = rt.msg( + rt.cls("NSDate"), + rt.sel("dateWithTimeIntervalSinceNow:"), + ctypes.c_void_p, + [ctypes.c_double], + float(max(seconds, 0.0)), + ) + mode = rt.msg( + rt.cls("NSString"), + rt.sel("stringWithUTF8String:"), + ctypes.c_void_p, + [ctypes.c_char_p], + _RUNLOOP_MODE, + ) + rt.msg( + run_loop, + rt.sel("runMode:beforeDate:"), + ctypes.c_bool, + [ctypes.c_void_p, ctypes.c_void_p], + mode, + date, + ) + except Exception: + logger.debug("overlay: run-loop pump failed", exc_info=True) + # Still yield the CPU, so a broken pump cannot turn into a spin. + time.sleep(max(seconds, 0.0)) + + # ── internals ── + + def _place(self, x_top: float, y_top: float) -> None: + """Place the glyph so its TIP lands on the given top-left point. + + Two conversions happen here and only here: + + * TOP-LEFT to BOTTOM-LEFT (``y_bottom = screen_height - y_top``), because + every other computer-use module speaks top-left and ``NSWindow``'s origin + is bottom-left; + * tip to window ORIGIN, by subtracting the glyph's hot spot. The hot spot + arrives in top-left glyph coordinates, so the vertical component is + measured from the glyph's top edge and the window origin sits + ``glyph_height - hotspot_y`` below the tip. + """ + if self._window is None: + return + px, py = self._clamp(x_top, y_top) + origin_x = px - self._hotspot_x + origin_y = (self._screen_height - py) - (self._glyph_height - self._hotspot_y) + if not (math.isfinite(origin_x) and math.isfinite(origin_y)): + # A NaN reaches AppKit as an un-placeable window rather than an error; + # dropping the frame is the honest behaviour. + return + try: + self._rt.msg( + self._window, + self._rt.sel("setFrameOrigin:"), + None, + [CGPoint], + CGPoint(origin_x, origin_y), + ) + except Exception: + logger.debug("overlay: place failed", exc_info=True) + + def _set_alpha(self, alpha: float) -> None: + """Set window alpha, clamped to ``[0, 1]``.""" + if self._window is None: + return + value = min(max(float(alpha), 0.0), 1.0) + try: + self._rt.msg( + self._window, self._rt.sel("setAlphaValue:"), None, [ctypes.c_double], value + ) + except Exception: + logger.debug("overlay: alpha failed", exc_info=True) + + def _clamp(self, x_top: float, y_top: float) -> tuple[float, float]: + """Keep a point on the measured main display. + + Deliberately forgiving about a non-finite input (mapped to the origin + corner) — a cosmetic renderer must not raise on a malformed command, and + the supervisor already validated the shape. + """ + x = float(x_top) if math.isfinite(x_top) else 0.0 + y = float(y_top) if math.isfinite(y_top) else 0.0 + max_x = max(self._screen_width - 1.0, 0.0) + max_y = max(self._screen_height - 1.0, 0.0) + return (min(max(x, 0.0), max_x), min(max(y, 0.0), max_y)) + + @property + def visible(self) -> bool: + """Whether the window is currently ordered in.""" + return self._visible + + +def parse_command(line: str) -> "dict[str, Any] | None": + """Parse one NDJSON command line, or ``None`` if it is not usable. + + Tolerant by design: this process's stdin is written by our own supervisor, but + a truncated write during a gateway crash is a real event and a malformed line + must be skipped rather than kill the renderer (which would leave a cursor on + screen — the exact failure the EOF exit exists to prevent). + """ + text = line.strip() + if not text: + return None + try: + payload = json.loads(text) + except (ValueError, TypeError): + logger.debug("overlay: unparseable command line") + return None + if not isinstance(payload, dict): + return None + kind = payload.get(OVERLAY_CMD_KEY) + if not isinstance(kind, str) or not kind: + return None + return payload + + +def _coerce_points(raw: Any) -> tuple[tuple[float, float], ...]: + """Coerce a command's ``points`` payload into finite float pairs. + + Every non-conforming entry is DROPPED rather than defaulted: a point silently + replaced by ``(0, 0)`` would fling the visible cursor to the corner of the + screen mid-animation, which is worse than a shorter path. + """ + if not isinstance(raw, list): + return () + out: list[tuple[float, float]] = [] + for item in raw: + if isinstance(item, (list, tuple)) and len(item) == 2: + first, second = item[0], item[1] + elif isinstance(item, dict): + first, second = item.get(OVERLAY_KEY_X), item.get(OVERLAY_KEY_Y) + else: + continue + if isinstance(first, bool) or isinstance(second, bool): + continue + if not isinstance(first, (int, float)) or not isinstance(second, (int, float)): + continue + x, y = float(first), float(second) + if not (math.isfinite(x) and math.isfinite(y)): + continue + out.append((x, y)) + return tuple(out) + + +def _coerce_number(raw: Any, default: float) -> float: + """A finite float from *raw*, else *default* (``bool`` is not a number here).""" + if isinstance(raw, bool) or not isinstance(raw, (int, float)): + return default + value = float(raw) + return value if math.isfinite(value) else default + + +def read_commands(stream: Any) -> "Any": + """Yield parsed commands from *stream* until EOF. + + Written as an explicit ``readline`` loop rather than ``for line in stream``: + file iteration buffers, so a command could sit unread until the buffer filled — + which for an animation means arriving after it was relevant. ``readline`` + returning ``""`` is EOF, and EOF is the signal that the parent is gone. + """ + while True: + try: + line = stream.readline() + except (OSError, ValueError): + # ValueError: the stream was closed under us during shutdown. + return + if not line: + return + if isinstance(line, bytes): + line = line.decode("utf-8", errors="replace") + command = parse_command(line) + if command is not None: + yield command + + +def _handle(window: CursorOverlayWindow, command: "dict[str, Any]") -> bool: + """Apply one command. Returns False when the renderer should exit.""" + kind = command.get(OVERLAY_CMD_KEY) + if kind == OVERLAY_CMD_QUIT: + return False + if kind == OVERLAY_CMD_HIDE: + window.hide() + return True + if kind == OVERLAY_CMD_MOVE: + points = _coerce_points(command.get(OVERLAY_KEY_POINTS)) + if not points: + # A bare {"type":"move","x":..,"y":..} is a legal one-point move: the + # supervisor uses it to park the cursor with no animation. + x = command.get(OVERLAY_KEY_X) + y = command.get(OVERLAY_KEY_Y) + if isinstance(x, (int, float)) and isinstance(y, (int, float)): + if not isinstance(x, bool) and not isinstance(y, bool): + points = _coerce_points([[x, y]]) + if points: + duration = int(_coerce_number(command.get(OVERLAY_KEY_MS), MIN_MOVE_DURATION_MS)) + window.move_along(points, duration) + return True + if kind == OVERLAY_CMD_CLICK: + x = _coerce_number(command.get(OVERLAY_KEY_X), math.nan) + y = _coerce_number(command.get(OVERLAY_KEY_Y), math.nan) + count = int(_coerce_number(command.get(OVERLAY_KEY_COUNT), 1.0)) + if math.isfinite(x) and math.isfinite(y): + window.pulse_click(x, y, count) + return True + logger.debug("overlay: ignoring unknown command %r", kind) + return True + + +def _idle_pump( + window: CursorOverlayWindow, stop: threading.Event, last_seen: "list[float]" +) -> None: + """Keep the run loop alive and auto-hide after an idle period. + + Runs on the READER thread's counterpart: the main thread owns AppKit, so this + helper is called FROM the main thread between commands. ``last_seen`` is a + one-element list rather than a scalar so the caller and this function share one + mutable cell without a class wrapper for two lines of state. + """ + if stop.is_set(): + return + window.pump(OVERLAY_IDLE_SLICE_SECS) + if window.visible and (time.monotonic() - last_seen[0]) >= OVERLAY_IDLE_HIDE_SECS: + # Backstop against a parent that stopped writing without closing the pipe. + # The primary anti-orphan guarantee is the EOF exit in ``main``. + window.hide() + + +def main(argv: "Sequence[str] | None" = None) -> int: + """Entry point: pump AppKit on the main thread, read stdin on a worker. + + The thread split is forced by AppKit: the run loop MUST be on the main thread, + and ``readline`` blocks. So stdin is read on a daemon thread that hands parsed + commands to the main thread through a lock-guarded list, and the main thread + alternates between draining that list and pumping the run loop. + + Returns 0 on every path — including "not macOS" and "AppKit unavailable". A + non-zero exit would make the supervisor log a child failure for a cosmetic + subsystem that correctly declined to run. + """ + del argv # No options: every parameter arrives on stdin. + if not platform_compat.IS_MACOS: + # Not an error: the supervisor already declines to spawn off macOS, and a + # user running the module by hand deserves a plain answer. + sys.stderr.write("cursor overlay is macOS-only; nothing to do\n") + return 0 + try: + runtime = ObjCRuntime() + except Exception: + logger.debug("overlay: ObjC runtime unavailable", exc_info=True) + sys.stderr.write("cursor overlay: AppKit unavailable\n") + return 0 + + window = CursorOverlayWindow(runtime) + pending: list[dict[str, Any]] = [] + lock = threading.Lock() + stop = threading.Event() + + def _reader() -> None: + try: + for command in read_commands(sys.stdin): + with lock: + pending.append(command) + finally: + # EOF (or a closed stream) means the parent is gone — THE anti-orphan + # guarantee. Setting the event is what makes the main loop tear the + # window down and return. + stop.set() + + thread = threading.Thread(target=_reader, name="overlay-stdin", daemon=True) + thread.start() + + # Announce readiness only after the window exists, so the supervisor can + # distinguish "spawned" from "actually able to draw". + ready = window.ensure() + try: + sys.stdout.write(f"{OVERLAY_READY_LINE} {1 if ready else 0}\n") + sys.stdout.flush() + except (OSError, ValueError): + logger.debug("overlay: ready line write failed", exc_info=True) + + last_seen = [time.monotonic()] + try: + while True: + with lock: + batch = pending[:] + del pending[:] + if batch: + last_seen[0] = time.monotonic() + for command in batch: + if not _handle(window, command): + return 0 + if not batch: + if stop.is_set(): + return 0 + _idle_pump(window, stop, last_seen) + finally: + # Always tear the window down: an orphan cursor on the user's screen is the + # single worst failure this process can produce. + window.close() + stop.set() + + +if __name__ == "__main__": # pragma: no cover - process entry point + logging.basicConfig(level=os.environ.get("KIROCREW_LOG_LEVEL", "WARNING")) + sys.exit(main(sys.argv[1:])) diff --git a/src/kiro_crew/computer_use/permissions.py b/src/kiro_crew/computer_use/permissions.py new file mode 100644 index 00000000000..6d5dba73c6c --- /dev/null +++ b/src/kiro_crew/computer_use/permissions.py @@ -0,0 +1,195 @@ +"""macOS TCC permission probing — **ADVISORY ONLY, NEVER A GATE**. + +This module exists to put a helpful hint in the Settings panel. It must never +decide whether an action proceeds, and no caller may treat a ``missing`` result as +"unavailable". That is not caution, it is a reproduced fact: during live probing +**both** grants reported ``missing`` while a full-fidelity window capture and a +complete accessibility tree came back successfully. + +The reason is how macOS attributes a TCC grant. The grant follows the +**responsible parent** of the process tree, not the process that asks. The +gateway spawns kiro-cli, which spawns the MCP sidecar, so the grant belongs to +whatever launched the tree — ``KiroCrew.app`` when packaged, ``Terminal.app`` for a +dev ``kirocrew gateway``, an IDE if launched from one. A probe inspects only its +own process's grant and therefore reports ``missing`` for a tree that is fully +authorized. :attr:`PermissionProbe.responsible_hint` names the process the user +should actually grant, because "grant Accessibility to KiroCrew" is unactionable +advice when the thing holding the grant is their terminal. + +Two related hazards this module is careful about: + +* ``CGRequestScreenCaptureAccess`` is **never** called and is not even bound in + :mod:`macos_ffi`. It pops a system dialog from the calling process, which from a + background sidecar is an unexplained prompt the operator cannot attribute to + anything they did. Only the ``Preflight`` variant is used. +* Ad-hoc re-signing anything in the process chain **voids an existing grant** + (observed permanently breaking Accessibility 3/3 times on a re-signed bundle). + ``packaging/resign-macos-libs.sh`` re-signs native libs, so a user whose + permissions "worked yesterday" may have been broken by an update rather than by + a setting. The hint text has to leave room for that. +""" + +from __future__ import annotations + +import logging +import os +import sys + +from kiro_crew import platform_compat +from kiro_crew.computer_use import macos_ffi +from kiro_crew.computer_use.types import ( + PERMISSION_GRANTED, + PERMISSION_MISSING, + PERMISSION_UNKNOWN, + PERMISSION_UNSUPPORTED, + PermissionProbe, +) + +logger = logging.getLogger(__name__) + +# Advisory copy for the Settings rows. Deliberately says "not detected" rather +# than "missing" or "denied": the probe genuinely cannot distinguish "not granted" +# from "granted to an ancestor we cannot see", and overclaiming here would send +# users to System Settings to fix something that is not broken. +HINT_RESPONSIBLE = ( + "macOS attributes Accessibility and Screen Recording grants to the process " + "that launched KiroCrew, not to KiroCrew itself. Grant them to {process} — " + "and note that 'not detected' does not always mean unavailable." +) +HINT_UNSUPPORTED = "computer use requires macOS; there is nothing to grant on this platform." +HINT_UNKNOWN = ( + "the permission state could not be read. Computer use may still work: grants " + "follow the launching process, which a probe cannot inspect." +) + +# System Settings deep links, for the Settings panel's grant buttons. +SETTINGS_URL_ACCESSIBILITY = ( + "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility" +) +SETTINGS_URL_SCREEN_RECORDING = ( + "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture" +) + +# Payload keys, so the handler, the CLI and the frontend share one spelling. +KEY_ACCESSIBILITY = "accessibility" +KEY_SCREEN_RECORDING = "screen_recording" +KEY_RESPONSIBLE_HINT = "responsible_hint" + +# Fallback label when the responsible process cannot be named. +_UNKNOWN_PROCESS = "the application that launched KiroCrew" + + +def probe() -> PermissionProbe: + """Probe the two TCC grants. ADVISORY. Never raises. + + Returns :data:`PERMISSION_UNSUPPORTED` off macOS, :data:`PERMISSION_UNKNOWN` + when the frameworks cannot be loaded or a probe call fails, and otherwise + ``granted``/``missing`` — with the standing caveat that ``missing`` is not + authoritative. + """ + if not platform_compat.IS_MACOS: + return PermissionProbe( + accessibility=PERMISSION_UNSUPPORTED, + screen_recording=PERMISSION_UNSUPPORTED, + responsible_hint=HINT_UNSUPPORTED, + ) + hint = HINT_RESPONSIBLE.format(process=responsible_process_name()) + try: + accessibility = ( + PERMISSION_GRANTED if macos_ffi.ax_is_process_trusted() else PERMISSION_MISSING + ) + except Exception: + logger.debug("AXIsProcessTrusted probe failed", exc_info=True) + accessibility = PERMISSION_UNKNOWN + hint = HINT_UNKNOWN + try: + screen = PERMISSION_GRANTED if macos_ffi.preflight_screen_capture() else PERMISSION_MISSING + except Exception: + logger.debug("CGPreflightScreenCaptureAccess probe failed", exc_info=True) + screen = PERMISSION_UNKNOWN + hint = HINT_UNKNOWN + return PermissionProbe( + accessibility=accessibility, screen_recording=screen, responsible_hint=hint + ) + + +def probe_dict() -> dict[str, str]: + """The probe as a plain dict, for the dashboard payload and ``doctor --json``. + + A separate function rather than ``dataclasses.asdict`` so the wire key names + are stated once, here, and cannot drift when a field is renamed. + """ + result = probe() + return { + KEY_ACCESSIBILITY: result.accessibility, + KEY_SCREEN_RECORDING: result.screen_recording, + KEY_RESPONSIBLE_HINT: result.responsible_hint, + } + + +def responsible_process_name() -> str: + """Best-effort name of the process that actually holds the TCC grants. + + Walks up the parent chain (through :mod:`platform_compat`, never a raw + ``/proc`` read or a ``ps`` shell-out) to the outermost ancestor we can still + identify, and reports its bundle display name when it has one. That ancestor + is what macOS considers responsible for the tree, so it is the process the user + must grant. + + Best-effort by design: the walk is bounded, every step tolerates failure, and + an unresolvable chain yields a generic phrase. A wrong hint is a cosmetic + problem; raising here would break the Settings page over a diagnostic. + """ + # Late import to avoid a module-scope cycle: apps_macos imports macos_ffi, + # which is fine, but permissions is imported by the driver that apps_macos + # also feeds, and keeping this local documents that the dependency is + # diagnostic-only. + from kiro_crew.computer_use import apps_macos + + pid = os.getpid() + best = "" + # Bounded ancestor walk. 8 levels is far more than the real chain + # (launcher -> gateway -> kiro-cli -> sidecar) and terminates regardless of + # what a hostile or unusual process tree reports. + for _ in range(8): + identity = apps_macos.resolve_identity(pid) + if identity.display_name: + best = identity.display_name + try: + parent = platform_compat.get_ppid(pid) + except Exception: + break + if not parent or parent <= 1 or parent == pid: + break + pid = parent + if best: + return best + # No bundle anywhere in the chain: a bare python/venv launch. Naming the + # executable is still more useful than nothing. + return os.path.basename(sys.executable) or _UNKNOWN_PROCESS + + +def settings_urls() -> dict[str, str]: + """System Settings deep links for the two grants (macOS only, else empty).""" + if not platform_compat.IS_MACOS: + return {} + return { + KEY_ACCESSIBILITY: SETTINGS_URL_ACCESSIBILITY, + KEY_SCREEN_RECORDING: SETTINGS_URL_SCREEN_RECORDING, + } + + +__all__ = [ + "HINT_RESPONSIBLE", + "HINT_UNKNOWN", + "HINT_UNSUPPORTED", + "KEY_ACCESSIBILITY", + "KEY_RESPONSIBLE_HINT", + "KEY_SCREEN_RECORDING", + "SETTINGS_URL_ACCESSIBILITY", + "SETTINGS_URL_SCREEN_RECORDING", + "probe", + "probe_dict", + "responsible_process_name", + "settings_urls", +] diff --git a/src/kiro_crew/computer_use/policy.py b/src/kiro_crew/computer_use/policy.py new file mode 100644 index 00000000000..11baf5eff83 --- /dev/null +++ b/src/kiro_crew/computer_use/policy.py @@ -0,0 +1,437 @@ +"""Target/input policy and egress redaction for computer use. + +Pure decision logic — no ctypes, no I/O, no platform calls. Every function here +is a *deny* gate: it either returns ``None`` (nothing to say) or a refusal +sentence. Nothing in this module can widen access. + +Why this module carries so much weight: computer use is structurally invisible +to every other security plane in KiroCrew. ``is_sensitive_path`` cannot see a +password field's ``AXValue``; the bash deny rules cannot see a keystroke posted +into a terminal window; the exfil URL scan cannot see a logged-in banking tab +rendered as pixels. The path matchers protect the filesystem the agent reaches +through *tools*; this module is the only thing standing between the agent and +the same data reached through *the operator's own windows*. +""" + +from __future__ import annotations + +from kiro_crew import security +from kiro_crew.computer_use.types import ( + CLICK_METHOD_ACCESSIBILITY, + CLICK_METHOD_APP_POST, + CLICK_METHOD_AUTO, + CLICK_METHOD_SKY_CLICK, + CLICK_METHODS, + DEFAULT_CLICK_METHOD, + ERR_POINT_REQUIRED, + ERR_SKY_CLICK_BUTTON, + ERR_UNKNOWN_CLICK_METHOD, + ERR_UNKNOWN_MOUSE_BUTTON, + MOUSE_BUTTON_LEFT, + MOUSE_BUTTONS, + REFUSAL_ACCESSIBILITY_NEEDS_INDEX, + REFUSAL_CLICK_TARGET_AMBIGUOUS, + REFUSAL_CLICK_TARGET_MISSING, + REFUSAL_DENIED_APP, + REFUSAL_SECURE_TARGET, + REFUSAL_TEXT_SENSITIVE, + AppRef, + DeniedApp, + ElementRec, + PolicyConfig, +) +from kiro_crew.platform import redact_via_context + +# ── Categories (stable ids: they appear in the dashboard payload and in tests) ── +CATEGORY_KIROCREW_SELF = "kirocrew_self" + +# ── The built-in target denylist (a FLOOR — code, not configuration) ── +# +# Matching is by bundle-id PREFIX (so a helper process under a blocked bundle is +# covered too) OR by a process-name SUBSTRING (the Windows and Linux drivers may +# only ever learn a process name, and macOS reports one for unbundled binaries). +# BOTH comparisons are case-insensitive on BOTH sides — entries below are written +# in whatever case Apple ships (``com.apple.userNotificationCenter``), and +# ``denied_rule_for`` lowercases the table entry as well as the subject. Comparing +# only a lowercased subject against a mixed-case entry made that one row +# unmatchable, which is a security row that reads as protection and does nothing. +# ``extra_denied_apps`` can only ADD to this list; there is deliberately no +# mechanism to remove an entry, so the floor cannot be edited away from the +# dashboard or by a prompt-injected agent. +# +# Honest scope statement, because a reviewer will find it otherwise: this list +# is *enumerable*, not provably complete. An app that embeds a shell without a +# recognizable bundle id — a terminal pane inside an IDE, an in-app JS console — +# is not covered. That is why the governance ``apps`` ruleset exists as the +# enterprise force-pin on top, and why the input-text scan below is a second, +# independent layer. +_DENIED_BUNDLE_PREFIXES: tuple[DeniedApp, ...] = ( + DeniedApp( + category=CATEGORY_KIROCREW_SELF, + # The ONE app that stays refused, and the only one that is not a judgement + # call about the operator's own machine. + # + # KiroCrew's Settings UI is where the computer-use primary enable lives, and + # that enable is on the keystone precisely so the AGENT cannot reach it (the + # keystone sits behind ``security._SENSITIVE_HOME_DIRS``). If the agent could + # DRIVE our own window, it would click that toggle itself — and every other + # security control on the same page — which routes around the keystone + # entirely. Refusing our own bundle is what keeps "the operator, out of band, + # is the only one who can widen this" true. + # + # Everything else that used to be here (terminals, password managers, System + # Settings, system auth dialogs) is deliberately GONE: on a personal machine + # the agent is trusted with the desktop, and a shipped list of "apps you may + # not automate" was both incomplete by construction and in the operator's way. + reason=( + "KiroCrew's own dashboard can change the agent's security settings, " + "which must only be done by the operator out-of-band" + ), + bundle_prefixes=("com.amazon.kiro.crew", "dev.kiro.crew"), + name_substrings=("kiro crew", "kirocrew"), + # The dashboard is ALSO reachable as a browser tab, where the app identity + # is Chrome's or Safari's and the two lists above cannot fire. The window + # title is the only signal that survives that hosting, so it carries the + # same rule. Substrings, not exact strings: the tab title takes a badge + # prefix ("(3) Kiro Crew") and popouts a "