Skip to content

Commit 0aeed87

Browse files
feat(website-browser): per-user CDP addresses, discovery and a WebSocket relay (#482)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent c655289 commit 0aeed87

16 files changed

Lines changed: 958 additions & 49 deletions

File tree

‎packages/agentenv-protocol/README.md‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,8 @@ app without one, so a server that mounted is a server whose tool calls are bound
150150
To answer a call before the tool runs, override `on_tool_call(self, context)`: return `None` to let
151151
it proceed, or an `mcp.types.CallToolResult` (`isError=True` for a refusal) to send that as the
152152
tool's reply. An `@extension` method may declare the same parameter; it is filled from the request's
153-
`AgentEnv-Role` header. Code that acts as a caller outside any dispatch, such as a test calling a
153+
`AgentEnv-Role` header. It may also declare a parameter annotated `starlette.requests.Request`, which
154+
gets the HTTP request itself, its headers say, and stays out of the advertised schema. Code that acts as a caller outside any dispatch, such as a test calling a
154155
handler directly or a seeding script, binds a context it builds:
155156
`with bound(ToolContext.as_caller("alice")): ...`.
156157

@@ -781,6 +782,11 @@ requests to the app's server, taking the control token:
781782
closes, the other then closed with its code; refused before they open without the token, or with no
782783
such app or one not answering. The `computer` extra brings `websockets` for this.
783784

785+
An app's `agent_routes` (`/cdp/` for a website browser) are path prefixes of its own server that
786+
agents reach this way without the control token: the driver reads them from the app's card
787+
(`computer-app/v1`'s `agent_routes`) and lets a call under one through, while the app checks it
788+
itself. A path with a `.` or `..` segment, escaped or not, never counts as under one.
789+
784790
`MailApp.declaration()` returns what the class declares, an `AppDeclaration`, which a `computer_app`
785791
env stores (`browser: true` for a website browser), and `python -m agentenv_protocol.computer._declare <folder> <file> <class>` prints it as
786792
JSON, or a driver's `DriverDeclaration`, without the class's own dependencies installed.
@@ -860,6 +866,22 @@ are the profile's template. The base class adds:
860866
- `files/choose {paths}`, files in `~/Uploads` for the file chooser open on it, which headless, and so
861867
alone, it intercepts;
862868
- `page/state`, the active tab's id, URL and title and the dialog or file chooser open on it.
869+
- `cdp {user}`, `user`'s CDP address once the browser is ready, `http_url` to discover from and
870+
`ws_url` to connect to, both at the address the call reached it at with a token agentenv mints in
871+
the path: the same one each time it's asked for. Through the driver they're under
872+
`/svc/<app>/cdp/<token>/`. `user` is a name the caller chooses, an agent's `deploy_agent` step id.
873+
The server keeps each token in memory and only its SHA-256, with its user and base address, in
874+
`.agentenv/cdp-tokens.json`, which code in the computer's session can read but not use; an address
875+
outlives a relaunch of the browser and of the server, and the computer's reset ends it.
876+
- **Its CDP relay**, at those addresses: `json/version` and `json/list` (also with a trailing slash,
877+
and `json`) answered from the browser, since a browser on a pipe has none, with its WebSocket at the
878+
address the client reached it at; and `devtools/browser`, where each WebSocket connection gets a
879+
browser-level session of its own over the pipe (`Target.attachToBrowserTarget`), with its answers
880+
and the events of its sessions and those it attaches, and no one else's. Only flattened sessions are
881+
served, as Playwright, Puppeteer and Browser Use use. A token agentenv didn't mint is refused, and
882+
when the browser goes, each connection is closed with 1012 so its client can connect again.
883+
`record` lists each connection as `cdp_connections`: its user, when it opened and when it closed.
884+
Puppeteer connects by `ws_url`: its `browserURL` drops the address's path.
863885
- **Its skill**: sites by their real names, files to upload in `~/Uploads` and downloads in
864886
`~/Downloads`, folders it makes.
865887

‎packages/agentenv-protocol/pyproject.toml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,11 @@ agent = [
3636
"uvicorn>=0.30",
3737
]
3838
# A computer app's own server: ComputerAppEnvironment serves its card and computer-app/v1 over mcp's FastMCP, on a Unix
39-
# socket through uvicorn.
39+
# socket through uvicorn, and a website browser's CDP relay over WebSockets.
4040
app = [
4141
"mcp>=1.25,<2",
4242
"uvicorn>=0.30",
43+
"websockets>=13",
4344
]
4445
# A computer driver: ComputerDriverEnvironment serves its tools over mcp's FastMCP, crops and encodes frames, records
4546
# the display as H.264 with PyAV (computer-recording/v1), and relays WebSockets to its apps on /svc/<app>/.

‎packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py‎

Lines changed: 33 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
from pydantic.fields import FieldInfo
2323
from starlette.requests import Request
2424
from starlette.responses import Response
25-
from starlette.routing import Route
25+
from starlette.routing import Route, WebSocketRoute
2626

2727
from .errors import AgentEnvError
2828
from .tool_access import DISABLE as TOOL_DISABLE, ENABLE as TOOL_ENABLE, PATHS as TOOL_ACCESS_PATHS, ToolAccess, caller_role
@@ -797,6 +797,19 @@ def _is_tool_context_annotation(annotation: Any) -> bool:
797797
return any(t is ToolContext for t in _unwrap_annotation(annotation))
798798

799799

800+
def _is_request_annotation(annotation: Any) -> bool:
801+
if isinstance(annotation, str):
802+
return Request.__name__ in _string_annotation_names(annotation)
803+
return any(t is Request for t in _unwrap_annotation(annotation))
804+
805+
806+
def _request_slot(fn: Callable) -> str | None:
807+
"""Name of an extension handler's parameter annotated ``Request``, which receives the HTTP request, or None."""
808+
hints = _resolved_hints(fn)
809+
return next((name for name, param in inspect.signature(fn).parameters.items() if name not in ("self", "cls")
810+
and _is_request_annotation(hints.get(name, param.annotation))), None)
811+
812+
800813
def _fastmcp_context_type() -> type | None:
801814
"""FastMCP's Context class when mcp is loaded, else None. mcp is an optional dependency, so it is
802815
looked up rather than imported; a Context-annotated handler has necessarily imported it already."""
@@ -1034,7 +1047,8 @@ def _schema_from_signature(fn: Callable) -> dict:
10341047
if name in ("self", "cls") or param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
10351048
continue
10361049
annotation = hints.get(name, param.annotation)
1037-
if _is_tool_context_annotation(annotation) or _is_fastmcp_context_annotation(annotation):
1050+
if _is_tool_context_annotation(annotation) or _is_fastmcp_context_annotation(annotation) or \
1051+
_is_request_annotation(annotation):
10381052
continue
10391053
prop = _schema_for_annotation(annotation)
10401054
if param.default is inspect.Parameter.empty:
@@ -1505,12 +1519,12 @@ def _coerce_params(fn: Callable, params: dict) -> dict:
15051519

15061520
def _extension_handler(fn: Callable, http_method: str) -> Callable:
15071521
"""REST handler for an @extension route: bind request params to the handler and return JSON. A
1508-
``ToolContext`` parameter is filled from the request headers and never bound
1509-
from the params."""
1510-
slot = _tool_context_slot(fn)
1522+
``ToolContext`` parameter is filled from the request headers, and a ``Request`` one with the request itself;
1523+
neither is bound from the params."""
1524+
slot, request_slot = _tool_context_slot(fn), _request_slot(fn)
15111525
signature = inspect.signature(fn)
1512-
if slot:
1513-
signature = signature.replace(parameters=[p for p in signature.parameters.values() if p.name != slot])
1526+
signature = signature.replace(parameters=[p for p in signature.parameters.values()
1527+
if p.name not in (slot, request_slot)])
15141528
op = fn.__name__ or "invoke"
15151529

15161530
async def handler(request: Request) -> Response:
@@ -1530,7 +1544,8 @@ async def handler(request: Request) -> Response:
15301544
except TypeError as e:
15311545
return _error_response(AgentEnvError("invalid_params", str(e)))
15321546
with bound(context_from_http(request, op, params)) as tool_context:
1533-
call_kwargs = {**params, slot: tool_context} if slot else params
1547+
call_kwargs = {**params, **({slot: tool_context} if slot else {}),
1548+
**({request_slot: request} if request_slot else {})}
15341549
try:
15351550
result = await _maybe_await(fn(**call_kwargs))
15361551
except AgentEnvError as e:
@@ -1574,6 +1589,16 @@ async def guarded(request: Request) -> Response:
15741589
return guard
15751590

15761591

1592+
def add_websocket_route(app: Any, path: str, endpoint: Callable) -> None:
1593+
"""``endpoint`` served at ``path`` as a WebSocket route of ``app``, an mcp or a standalone fastmcp FastMCP, which
1594+
have no API for one."""
1595+
for routes in ("_custom_starlette_routes", "_additional_http_routes"):
1596+
if isinstance(getattr(app, routes, None), list):
1597+
getattr(app, routes).append(WebSocketRoute(path, endpoint))
1598+
return
1599+
raise TypeError(f"WebSocket routes go on an mcp or a fastmcp FastMCP, not a {type(app).__name__}")
1600+
1601+
15771602
def _tool_access_handler(access: ToolAccess, action: str) -> Callable:
15781603
async def handler(request: Request) -> Response:
15791604
raw = await request.body()
Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
"""The CDP addresses a website browser hands agents: a token agentenv mints for each user, of which only a hash is kept
2+
where the computer's session can read it, discovery answered from the browser, and a relay that gives each WebSocket
3+
connection its own browser-level session over the browser's one debugging pipe."""
4+
from __future__ import annotations
5+
6+
import asyncio
7+
import contextlib
8+
import hashlib
9+
import json
10+
import logging
11+
import os
12+
import secrets
13+
from dataclasses import dataclass
14+
from datetime import datetime, timezone
15+
from pathlib import Path
16+
from typing import TYPE_CHECKING, Optional
17+
18+
from starlette.requests import Request
19+
from starlette.websockets import WebSocket, WebSocketDisconnect
20+
21+
if TYPE_CHECKING:
22+
from .website_browser import DevToolsPipe
23+
24+
logger = logging.getLogger(__name__)
25+
26+
#: How many random bytes a token holds.
27+
TOKEN_BYTES = 32
28+
#: The route agents reach the relay under, through the driver's /svc/<app>/.
29+
ROUTE = "/cdp/"
30+
#: The close code and reason a connection gets when its browser goes away, which a client may connect again after.
31+
RELAUNCHED = (1012, "the browser relaunched: connect again")
32+
#: The close code and reason a connection gets when its client falls behind by more than ``OUTBOX_LIMIT_BYTES``.
33+
TOO_SLOW = (1013, "the client fell too far behind: connect again")
34+
#: How much a connection's client may have waiting to be sent to it before it is closed; a single answer bigger than
35+
#: this still goes when nothing else waits.
36+
OUTBOX_LIMIT_BYTES = 64 << 20
37+
#: CDP's own answer to a command for a session the connection doesn't have.
38+
NO_SESSION = {"code": -32001, "message": "Session with given id not found."}
39+
40+
41+
@dataclass
42+
class Address:
43+
"""A user's CDP address: ``http_url``, the base clients discover from, and ``ws_url``, the browser's WebSocket."""
44+
45+
user: str
46+
http_url: str
47+
ws_url: str
48+
49+
50+
def reached(request: Request) -> str:
51+
"""The address a request reached the browser at, scheme, host and path prefix: what the driver forwarded, else the
52+
request's own."""
53+
proto = request.headers.get("x-forwarded-proto") or request.url.scheme
54+
host = request.headers.get("x-forwarded-host") or request.headers.get("host") or request.url.netloc
55+
return f"{proto}://{host}{request.headers.get('x-forwarded-prefix', '')}".rstrip("/")
56+
57+
58+
def websocket_form(url: str) -> str:
59+
return "wss://" + url.removeprefix("https://") if url.startswith("https://") else \
60+
"ws://" + url.removeprefix("http://")
61+
62+
63+
def _hash(token: str) -> str:
64+
return hashlib.sha256(token.encode()).hexdigest()
65+
66+
67+
def _address(user: str, base: str, token: str) -> Address:
68+
root = f"{base}{ROUTE}{token}"
69+
return Address(user, f"{root}/", f"{websocket_form(root)}/devtools/browser")
70+
71+
72+
class Tokens:
73+
"""The tokens minted, one per user: in memory, so asking twice gives the same address, and in ``path`` only by
74+
hash, with the user and the base address, so an address still works after the server restarts while the
75+
computer's session, which can read the file, can't use one."""
76+
77+
def __init__(self, path: Path) -> None:
78+
self.path = path
79+
self._minted: dict[str, Address] = {}
80+
try:
81+
stored = json.loads(path.read_text())
82+
self._known = {digest: (entry["user"], entry["base"]) for digest, entry in stored.items()}
83+
except (OSError, ValueError, TypeError, KeyError):
84+
self._known = {}
85+
86+
def mint(self, user: str, base: str) -> Address:
87+
"""``user``'s address under ``base``: the one minted before, else a new one."""
88+
if user not in self._minted:
89+
token = secrets.token_urlsafe(TOKEN_BYTES)
90+
self._minted[user] = _address(user, base, token)
91+
self._known[_hash(token)] = (user, base)
92+
self._save()
93+
return self._minted[user]
94+
95+
def address(self, token: str) -> Optional[Address]:
96+
"""The address ``token`` was minted for, or None for one that wasn't."""
97+
known = self._known.get(_hash(token))
98+
return _address(*known, token) if known is not None else None
99+
100+
def _save(self) -> None:
101+
self.path.parent.mkdir(parents=True, exist_ok=True)
102+
part = self.path.with_name(f".{self.path.name}.part")
103+
part.write_text(json.dumps({digest: {"user": user, "base": base}
104+
for digest, (user, base) in self._known.items()}))
105+
os.replace(part, self.path)
106+
107+
108+
class _Connection:
109+
"""One client's WebSocket, its sessions over the pipe (its browser-level one and those it attached), its unanswered
110+
commands, and what waits to be sent to it."""
111+
112+
def __init__(self, websocket: WebSocket, user: str, session: str) -> None:
113+
self.websocket, self.user, self.session = websocket, user, session
114+
self.sessions = {session}
115+
self.commands: set[asyncio.Future] = set()
116+
self.outbox: asyncio.Queue[str | tuple[int, str]] = asyncio.Queue()
117+
self.waiting_bytes = 0
118+
self.closing = False
119+
120+
121+
class Relay:
122+
"""Each WebSocket connection given its own browser-level session over ``devtools``, its commands sent on it and the
123+
answers and events of its sessions sent back, so several clients share one browser; and a record of who connected
124+
when."""
125+
126+
def __init__(self, devtools: DevToolsPipe) -> None:
127+
self.devtools = devtools
128+
self.connections: list[dict] = []
129+
self._by_session: dict[str, _Connection] = {}
130+
self._open: set[_Connection] = set()
131+
devtools.listen(self._event)
132+
133+
async def serve(self, websocket: WebSocket, user: str) -> None:
134+
"""``websocket``, a client of ``user``'s, relayed over a browser-level session of its own until either ends."""
135+
try:
136+
session = (await self.devtools.call("Target.attachToBrowserTarget"))["sessionId"]
137+
except Exception as e:
138+
await websocket.close(code=1013, reason=f"the browser isn't answering: {type(e).__name__}")
139+
return
140+
connection = _Connection(websocket, user, session)
141+
self._by_session[session] = connection
142+
self._open.add(connection)
143+
record = {"user": user, "opened_at": _now()}
144+
self.connections.append(record)
145+
await websocket.accept()
146+
sending = asyncio.ensure_future(self._send(connection))
147+
try:
148+
while True:
149+
message = await websocket.receive()
150+
if message["type"] == "websocket.disconnect":
151+
break
152+
text = message.get("text") if message.get("text") is not None else message["bytes"].decode()
153+
command = asyncio.ensure_future(self._command(connection, json.loads(text)))
154+
connection.commands.add(command)
155+
command.add_done_callback(connection.commands.discard)
156+
except (WebSocketDisconnect, ValueError):
157+
pass
158+
finally:
159+
record["closed_at"] = _now()
160+
connection.closing = True
161+
self._end(connection)
162+
for task in [*connection.commands, sending]:
163+
task.cancel()
164+
await asyncio.gather(*connection.commands, sending, return_exceptions=True)
165+
with contextlib.suppress(Exception):
166+
await self.devtools.call("Target.detachFromTarget", {"sessionId": session})
167+
168+
def ended(self) -> None:
169+
"""The browser has gone: every connection is closed with a reason a client may connect again after."""
170+
for connection in list(self._open):
171+
self._close(connection, RELAUNCHED)
172+
173+
async def _command(self, connection: _Connection, message: dict) -> None:
174+
session = message.get("sessionId") or connection.session
175+
if session not in connection.sessions:
176+
answer = {"error": NO_SESSION}
177+
else:
178+
try:
179+
answer = await self.devtools.request(message.get("method", ""), message.get("params") or {},
180+
session_id=session)
181+
except ConnectionError as e:
182+
answer = {"error": {"code": -32000, "message": str(e)}}
183+
reply = {"id": message.get("id"), **{key: answer[key] for key in ("result", "error") if key in answer}}
184+
if message.get("sessionId"):
185+
reply["sessionId"] = message["sessionId"]
186+
self._put(connection, reply)
187+
188+
def _event(self, message: dict) -> None:
189+
connection = self._by_session.get(message.get("sessionId"))
190+
if connection is None:
191+
return
192+
params = message.get("params") or {}
193+
if message.get("method") == "Target.attachedToTarget" and params.get("sessionId"):
194+
connection.sessions.add(params["sessionId"])
195+
self._by_session[params["sessionId"]] = connection
196+
elif message.get("method") == "Target.detachedFromTarget" and params.get("sessionId"):
197+
connection.sessions.discard(params["sessionId"])
198+
self._by_session.pop(params["sessionId"], None)
199+
if message["sessionId"] == connection.session:
200+
message = {key: value for key, value in message.items() if key != "sessionId"}
201+
self._put(connection, message)
202+
203+
def _put(self, connection: _Connection, message: dict) -> None:
204+
"""``message`` waiting to be sent to ``connection``'s client, which is closed instead, with what waited dropped,
205+
once it has fallen too far behind."""
206+
if connection.closing:
207+
return
208+
text = json.dumps(message)
209+
if connection.waiting_bytes and connection.waiting_bytes + len(text) > OUTBOX_LIMIT_BYTES:
210+
self._close(connection, TOO_SLOW)
211+
return
212+
connection.waiting_bytes += len(text)
213+
connection.outbox.put_nowait(text)
214+
215+
def _close(self, connection: _Connection, close: tuple[int, str]) -> None:
216+
"""``connection`` closed with ``close`` once what's being sent to it has gone, and nothing else sent."""
217+
connection.closing = True
218+
self._end(connection)
219+
while not connection.outbox.empty():
220+
connection.outbox.get_nowait()
221+
connection.waiting_bytes = 0
222+
connection.outbox.put_nowait(close)
223+
224+
def _end(self, connection: _Connection) -> None:
225+
self._open.discard(connection)
226+
for session in connection.sessions:
227+
self._by_session.pop(session, None)
228+
229+
async def _send(self, connection: _Connection) -> None:
230+
with contextlib.suppress(WebSocketDisconnect, RuntimeError):
231+
while True:
232+
message = await connection.outbox.get()
233+
if isinstance(message, tuple):
234+
await connection.websocket.close(code=message[0], reason=message[1])
235+
return
236+
connection.waiting_bytes -= len(message)
237+
await connection.websocket.send_text(message)
238+
239+
240+
def _now() -> str:
241+
return datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z")

0 commit comments

Comments
 (0)