Skip to content

Commit e22691e

Browse files
feat(drivers): cua-driver 0.44 on Hyprland, keys to the focused window and text to the desktop (#559)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 6fe61a8 commit e22691e

5 files changed

Lines changed: 10686 additions & 45 deletions

File tree

‎src/agent_env/drivers/README.md‎

Lines changed: 26 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -81,32 +81,44 @@ cua-driver's UI Automation tree, whose walk of it fails.
8181
On a Hyprland desktop (Wayland), as Omarchy's is, cua-driver runs its Wayland backend (`CUA_DRIVER_RS_ENABLE_WAYLAND=1`)
8282
and sends input through the seats of Hyprland's `cua-hyprland-plugin`. Hyprland's session turns this on
8383
(`HYPRLAND_INSTANCE_SIGNATURE`), not any Wayland one, since the driver reaches Hyprland's IPC; `supports` names no
84-
Wayland desktop yet, so gate 1 doesn't match it to one. The daemon starts once the compositor's socket answers, on a
85-
socket in the session user's runtime folder. What the driver asks Hyprland (`hyprctl -j`) is in `hyprland.py`, beside
86-
`driver.py`. Keys and text go to the window Hyprland has focused, in the foreground, and to the desktop with nothing
87-
focused; clicks, drags and scrolls stay desktop-scope. `computer-ready` also checks that the plugin is loaded, enabled,
84+
Wayland desktop yet, so gate 1 doesn't match it to one. It runs cua-driver 0.44.0, its own pin, since 0.38.0 refuses
85+
background input there, the delivery cua-driver's own tools default to: `computer-ready` checks its wheel's executable
86+
(`hyprland/amd64` in `EXECUTABLES`) and the tools `mcp-hyprland.json` pins, while `requirements.txt` keeps 0.38.0 for
87+
the other desktops, so a Hyprland computer installs 0.44.0 itself. The daemon starts once the compositor's socket
88+
answers, on a socket in the session user's runtime folder. What the driver asks Hyprland (`hyprctl -j`) is in
89+
`hyprland.py`, beside `driver.py`. Text goes to the desktop in the foreground, where Hyprland's shortcuts, a menu
90+
holding the keyboard and non-ASCII text all work. A key goes to the window Hyprland has focused, in the foreground,
91+
where Backspace, the arrows and Home/End edit text as a person's keys do, which at the desktop they don't, pressed as on
92+
a US keyboard, the only layout the plugin takes there, so a shifted character goes as Shift and the key it's on. Super
93+
and keys held with it, Hyprland's own shortcuts, go to the desktop, as does a key with nothing focused and one the
94+
window path refuses because a layer such as Omarchy's menu holds the keyboard, punctuation there by its keysym name:
95+
given the character, cua-driver falls back to libei, which can hang its input worker. Clicks, drags and scrolls stay
96+
desktop-scope. Keys aimed at a window need the seat's keymap to be the US layout's, and an input method such as
97+
Omarchy's fcitx5 takes on the keymap that desktop text uploads, after which cua-driver refuses them
98+
(`unsupported_layout`): the driver then puts the layout back with a chord of the layout's own keys at the desktop,
99+
Super+Ctrl+Alt+Shift+F12, which it binds to nothing in Hyprland first (`hyprctl eval`), so Hyprland keeps it from every
100+
window, and sends the key again. It presses the chord only while its own bind alone has it: with the chord bound to
101+
anything else, the key fails as cua-driver refused it. `computer-ready` also checks that the plugin is loaded, enabled,
88102
built for this Hyprland and taking input (`transport.ready`), and every input, a `computer_*` operation or one of
89103
cua-driver's own tools that isn't read-only, checks it again first and fails as `device_unavailable`, sending nothing,
90104
when it isn't. The runtime runs ready checks only until they first pass, and a Hyprland restarted after a crash runs in
91105
safe mode with no plugins, where input is accepted but never arrives. A plugin loaded again into a running Hyprland has
92106
no input seat until the desktop restarts, and a driver that outlives its Hyprland can't reach the next one, which has a
93-
new instance signature, so the driver's services have to restart with the graphical session. Keys aimed at a window
94-
skip Hyprland's own shortcuts, such as Super+2, and are refused while a layer such as Omarchy's menu holds the keyboard,
95-
and non-ASCII text aimed at one is refused; sent to the desktop in the foreground, all three work, but Backspace, the
96-
arrows and Home/End do nothing in a text box, so neither way alone covers both.
107+
new instance signature, so the driver's services have to restart with the graphical session.
97108

98109
To move it to another cua-driver version:
99110

100111
1. Pin the version in `requirements.txt`.
101112
2. Regenerate `mcp.json` from what that version's MCP server, `cua-driver mcp`, says at connect and lists: its
102113
`instructions`, each tool's `name`, `description`, `inputSchema`, `annotations` and `outputSchema`, and each
103114
resource's `uri`, `name`, `description` and `mimeType`. That takes a running daemon, so do it on a Linux test
104-
machine, `mcp-macos.json` the same way in a macOS computer, and `mcp-windows.json` in a Windows computer's signed-in
105-
session, on the driver's venv, since `cua-driver mcp` refuses Session 0, where ssh lands.
106-
3. Set `EXECUTABLES` in `driver.py` to the sha256 of `cua_driver/bin/cua-driver` in each arch's wheel, and of
107-
CuaDriver.app's executable for `macos/arm64`, and the release tarball's URL and sha256 in `env.toml`'s
108-
`[install.macos]` and its folder in `install-app.sh`; for `windows/amd64`, the zip's `cua-driver.exe`, its URL and
109-
sha256 in `[install.windows]` and its folder in `install-windows.ps1`.
115+
machine, `mcp-hyprland.json` the same way on a Hyprland desktop at its own pin, `mcp-macos.json` in a macOS
116+
computer, and `mcp-windows.json` in a Windows computer's signed-in session, on the driver's venv, since
117+
`cua-driver mcp` refuses Session 0, where ssh lands.
118+
3. Set `EXECUTABLES` in `driver.py` to the sha256 of `cua_driver/bin/cua-driver` in each arch's wheel, and in the wheel
119+
of Hyprland's own pin for `hyprland/amd64`, and of CuaDriver.app's executable for `macos/arm64`, and the release
120+
tarball's URL and sha256 in `env.toml`'s `[install.macos]` and its folder in `install-app.sh`; for `windows/amd64`,
121+
the zip's `cua-driver.exe`, its URL and sha256 in `[install.windows]` and its folder in `install-windows.ps1`.
110122
4. Run `tst/unit/drivers/cua_driver_test.py`, then `tst/integration/computer/cua_driver_test.py` on Linux, where
111123
`computer-ready` names any tool or executable that doesn't match its pin.
112124

‎src/agent_env/drivers/envs/cua-driver/driver.py‎

Lines changed: 54 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,9 @@
99
cua-driver starts again after ending it for idling, where a named one stays ended. A call the daemon can't take, as
1010
while it restarts, fails as unavailable; input whose reply is lost fails with its outcome unknown, since it may have
1111
arrived. It needs ``agentenv-framework-protocol[computer]`` and ``cua-driver`` where it runs. On a Hyprland desktop
12-
(Wayland) it runs cua-driver's Wayland backend, sends keys and text to the focused window in the foreground, and is
13-
ready, and takes input, only while Hyprland has cua-driver's plugin loaded, asking Hyprland through hyprland.py."""
12+
(Wayland) it runs cua-driver's Wayland backend at its own pin, sends text to the desktop and keys to the focused
13+
window in the foreground, Super and a key a menu holds the keyboard from going to the desktop, and is ready, and takes
14+
input, only while Hyprland has cua-driver's plugin loaded, asking Hyprland through hyprland.py."""
1415
from __future__ import annotations
1516

1617
import asyncio
@@ -77,18 +78,20 @@
7778
SOCKET = "/run/agentenv/cua-driver.sock"
7879
#: What cua-driver's own MCP server (``cua-driver mcp``) says at connect and lists at the pinned version on each OS: its
7980
#: instructions, its tools, which the running daemon must list the same, and its skill resources.
80-
MANIFEST = Path(__file__).with_name("mcp-macos.json" if MACOS else "mcp-windows.json" if WINDOWS else "mcp.json")
81+
MANIFEST = Path(__file__).with_name("mcp-macos.json" if MACOS else "mcp-windows.json" if WINDOWS else
82+
"mcp-hyprland.json" if HYPRLAND else "mcp.json")
8183
#: Where the release's zip is unpacked on Windows, beside this file.
8284
WINDOWS_EXECUTABLE = Path(__file__).with_name("cua-driver") / "cua-driver.exe"
8385
#: CuaDriver.app's executable, which macOS runs the daemon as.
8486
APP_EXECUTABLE = Path("/Applications/CuaDriver.app/Contents/MacOS/cua-driver")
85-
#: The sha256 of the cua-driver executable each arch's pinned wheel installs, on macOS CuaDriver.app's, and on Windows
86-
#: the release zip's.
87+
#: The sha256 of the cua-driver executable each arch's pinned wheel installs, on macOS CuaDriver.app's, on Windows the
88+
#: release zip's, and on Hyprland its own pin's wheel's, 0.44, which delivers background input there.
8789
EXECUTABLES = {
8890
"amd64": "bf57b2f3778a0a7daa9388e241f20ce948b32a7a2f4dd102e497f34e54137934",
8991
"arm64": "26e9eee8165e9ce7b614b5e3d68e5d963231385a82b65236406718b5ce1ff936",
9092
"macos/arm64": "b4d974a9c1c92a700cd7be7ede987aea4a1e6835e37039ce3d628b71bd5f0ca8",
9193
"windows/amd64": "ffa58d88779ee219304b0c307f4cfa970fe822f830c6ecf9254eda5c29e22bf2",
94+
"hyprland/amd64": "f4259d5135d23d4d03968e074d144a10ece6305ca28ef48edfb3ef5c9e724228",
9295
}
9396
#: How the daemon delivers the pointer's actions: in the foreground on macOS, where it doesn't by default.
9497
FOREGROUND = {"delivery_mode": "foreground"} if MACOS else {}
@@ -140,6 +143,7 @@ def __init__(self) -> None:
140143
self._skill: tuple[str, str, dict[str, str]] | None = None
141144
self._texts: dict[str, str] | None = None
142145
self._keymap: _Keymap | None = None
146+
self._reset_bind: dict | None = None
143147

144148
def forwarded_tools(self) -> list[ForwardedTool]:
145149
return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], inputSchema=t["inputSchema"]),
@@ -209,13 +213,53 @@ async def scroll_at(self, x, y, direction, amount, modifiers) -> None:
209213
async def type_in(self, text) -> None:
210214
if not MACOS and not WINDOWS and not HYPRLAND and any(ord(ch) >= LATIN_1 for ch in text):
211215
await asyncio.to_thread(self._hold, text)
212-
await self._send("type_text", text=text, **(await hyprland.focused() if HYPRLAND else {}))
216+
await self._send("type_text", text=text, **(hyprland.DESKTOP if HYPRLAND else {}))
213217

214218
@press
215219
async def press_keys(self, keys) -> None:
216220
*held, key = keys
217-
await self._send("press_key", modifiers=_modifiers(held), key=MODIFIERS.get(key) or NAMED.get(key, key),
218-
**(await hyprland.focused() if HYPRLAND else {}))
221+
modifiers, name = _modifiers(held), MODIFIERS.get(key) or NAMED.get(key, key)
222+
if HYPRLAND:
223+
await self._press_on_hyprland(modifiers, name)
224+
else:
225+
await self._send("press_key", modifiers=modifiers, key=name)
226+
227+
async def _press_on_hyprland(self, modifiers: list[str], key: str) -> None:
228+
"""A key on Hyprland: to the focused window as a person presses it on a US keyboard, where it edits text as a
229+
person's does, unless it's Super or held with it, which is Hyprland's own, or nothing is focused, or a layer
230+
such as a menu holds the keyboard and refuses it there; then to the desktop, punctuation by its keysym name."""
231+
if key != "super" and "super" not in modifiers and (window := await hyprland.focused()):
232+
try:
233+
return await self._to_window({**hyprland.on_us_keyboard(modifiers, key), **window})
234+
except DeliveryFailed as e:
235+
if hyprland.GRABBED not in e.message:
236+
raise
237+
await self._send("press_key", modifiers=modifiers, key=hyprland.PUNCTUATION.get(key, key),
238+
**hyprland.DESKTOP)
239+
240+
async def _to_window(self, pressed: dict) -> None:
241+
"""A key aimed at a window, the US layout's keymap put back first when cua-driver refuses it for the seat's
242+
keymap, which an input method changes after desktop text, by a chord bound to nothing but this driver's bind,
243+
which Hyprland keeps from every window."""
244+
try:
245+
return await self._send("press_key", **pressed)
246+
except DeliveryFailed as e:
247+
if hyprland.UNLAID not in e.message:
248+
raise
249+
if (bind := await hyprland.reset_bind(self._reset_bind)) is None:
250+
raise DeliveryFailed(e.message, hint=f"Hyprland has {hyprland.RESET_BIND}, which puts the US keymap back, "
251+
"bound to something this driver didn't bind, or it couldn't bind "
252+
"it, so the driver didn't press it") from e
253+
self._reset_bind = bind
254+
await self._send("press_key", **hyprland.RESET, **hyprland.DESKTOP)
255+
deadline = time.monotonic() + hyprland.RESET_S
256+
while True:
257+
try:
258+
return await self._send("press_key", **pressed)
259+
except DeliveryFailed as e:
260+
if hyprland.UNLAID not in e.message or time.monotonic() > deadline:
261+
raise
262+
await asyncio.sleep(hyprland.RESET_POLL_S)
219263

220264
async def _forward(self, tool: str, acts: bool, /, **arguments: Any) -> CallToolResult:
221265
"""A call to one of cua-driver's own tools in its caller's session, answered as cua-driver answers it; one that
@@ -275,7 +319,8 @@ async def _pinned_checks(self) -> list[ReadyCheck]:
275319
return [ReadyCheck("cua-driver", False, "its daemon isn't answering yet")]
276320
arch = self.platform.arch or ""
277321
digest = await asyncio.to_thread(_executable_sha256)
278-
expected = EXECUTABLES.get(f"macos/{arch}" if MACOS else f"windows/{arch}" if WINDOWS else arch)
322+
expected = EXECUTABLES.get(f"macos/{arch}" if MACOS else f"windows/{arch}" if WINDOWS else
323+
f"hyprland/{arch}" if HYPRLAND else arch)
279324
running = {t["name"]: t["input_schema"] for t in listed}
280325
pinned = {t["name"]: t["inputSchema"] for t in _manifest()["tools"]}
281326
differ = sorted(name for name in running.keys() | pinned.keys() if running.get(name) != pinned.get(name))

0 commit comments

Comments
 (0)