Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md

Large diffs are not rendered by default.

41 changes: 30 additions & 11 deletions packages/agentenv-protocol/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1194,17 +1194,36 @@ are the profile's template. The base class adds:
The server keeps each token in memory and only its SHA-256, with its user and base address, in
`.agentenv/cdp-tokens.json`, which code in the computer's session can read but not use; an address
outlives a relaunch of the browser and of the server, and the computer's reset ends it.
- **`browser-recording/v1`** on its card, framework-only, with `max_duration_s`, `max_frames` and `frames`:
`start {request_id, upload, frames?}` records under the write namespace grant `upload`, a start again with the
same `request_id` answering the recording it began; `end {recording_id, upload?, reason?}` ends it, under a fresh
grant when one is given, and answers its manifest; `end_all {reason?}` ends every open one. With `frames`, a frame
is a JPEG of the tab an agent last acted in, else of the active tab: one at the start, one after each burst of an
agent's acting entries (CDP commands and MCP calls, not agentenv's own) has gone quiet for 0.5 s and its tab's page
has loaded (up to 5 s), and one at the end, each uploaded as it's taken under `<recording_id>/frames/` and keyed in
the manifest to the entries it follows (`after`); a frame it couldn't take is in the manifest with its `error`. At
the end it uploads the log's lines since its start, actions and tab events, as `log.jsonl`, and `manifest.json`
(`BrowserRecordingManifest`). `start_browser_recording` and `end_browser_recording` run it, registering the frames,
log and manifest as one file artifact universe, and the end fails the run when its log or a frame is missing.
- **`browser-recording/v1`** on its card, framework-only: one recording per context, a lease's or the default
one's. Its params name what it records (`video`, `frames`, `network`, `context`) and its limits (`fps`,
`max_duration_s`, `max_frames`, `max_open`, `max_children`, `max_objects`).
`start {request_id, upload, context?, fps?, network?, frames?, max_edge?}` records the lease `context` names, else
the default context, under the write namespace grant `upload`, a start again with the same `request_id` answering
the recording it began; `end {recording_id, upload?, reason?}` ends it, under a fresh grant when one is given, and
answers its manifest (`BrowserRecordingManifest`); `end_all {reason?}` ends every open one. A recording has:
- **a video** of the tab its context's agent is on, at `fps` (10 by default, 15 at most), encoded as
`computer-recording/v1`'s is and uploaded a segment at a time under `<recording_id>/video/`. It follows the tab an
agent last acted on in that context (CDP commands and MCP calls, not agentenv's own), a tab that one opens as
soon as it opens, and once it closes the tab followed before it, else the context's newest; it's blank while the
context has no tab. The browser's own connection screencasts that tab, and captures it instead while it sends no
frame, as a tab behind another in its window does; `video.followed` says which tab showed when, and how.
- **its log**: at the end, the lines of its context since its start, actions and tab events on its tabs and a
scoped connection's actions on none, as `log.jsonl`.
- **the network log**, by default when the browser keeps one, as a browser alone does and one in a desktop
computer doesn't (`network` false leaves it out, true refuses a browser with none): the browser's whole log at
the end as `network.har`, so the start tabs' loads are in it, every context's requests among them; the manifest
marks it browser-wide, with the log lines written while it ran (`during`) and the leases open meanwhile.
- **frames**, with `frames` (off by default): a JPEG of the tab the video follows at the start, after each burst of
an agent's acting entries has gone quiet for 0.5 s and its page has loaded (up to 5 s), and at the end, under
`<recording_id>/frames/`, keyed to the entries it follows (`after`); a frame it couldn't take is kept with its
`error`.
- **children**: a context an agent makes over an address scoped to the recorded context is recorded with it, its
own video, log and frames under `<recording_id>/contexts/<label>/`, until its context or the recording ends.
A lease's recording ends with the lease (`context_ended`). The server encodes with PyAV and Pillow, which agentenv's
chromium installs through its own requirements; an app without them refuses a start. `start_browser_recording` and
`end_browser_recording` run it, registering everything it uploaded as one file artifact universe, and the end fails
the run when any of it is missing. The page driver's `computer-recording/v1` is a different recording: the page tab
as pixel tools see it, across contexts.
- **Its CDP relay**, at those addresses: `json/version` and `json/list` (also with a trailing slash,
and `json`) answered from the browser, since a browser on a pipe has none, with its WebSocket at the
address the client reached it at; and `devtools/browser`, where each WebSocket connection gets a
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""Recording a computer's display, computer-recording/v1: the base class runs each recording from the driver's
``@record`` stream, or by polling ``@capture`` where the driver has none, encodes frames as H.264 segments, uploads
each segment as it closes and seals the recording at ``end`` with its manifest."""
each segment as it closes and seals the recording at ``end`` with its manifest. ``Video`` is that recording, from any
source of frames, which a website browser's recordings (browser-recording/v1) use too."""
from __future__ import annotations

import asyncio
Expand Down Expand Up @@ -135,7 +136,7 @@ class Recordings:

def __init__(self, runtime: ComputerRuntime) -> None:
self.runtime = runtime
self.running: dict[str, _Recording] = {}
self.running: dict[str, Video] = {}
self.ended: OrderedDict[str, dict] = OrderedDict()
self.started: OrderedDict[str, dict] = OrderedDict()
self._lock = asyncio.Lock()
Expand All @@ -161,10 +162,13 @@ async def start(self, request_id: str, scope: str, window: Optional[str], fps: O
root = Path.home() / RECORDINGS_DIR
hints = dataclasses.replace(hints, directory=root / recording_id)
hints.directory.mkdir(parents=True, exist_ok=True)
recording = _Recording(recording_id, hints, features, root, grant, self.runtime)
streams, runtime = self.runtime.has("record"), self.runtime
cursor = hints.cursor if streams else hints.cursor and runtime.has("pointer")
source = (lambda hints: runtime.stream(hints)) if streams else \
(lambda hints: _poll(runtime, hints, cursor))
recording = Video(recording_id, hints, features, root, grant, source, streams=streams, cursor=cursor)
self.running[recording_id] = recording
recording.task = asyncio.create_task(recording.run())
recording.timer = asyncio.get_running_loop().call_later(features.max_duration_s, recording.expire)
recording.begin()
answer = {"recording_id": recording_id, "started_at": _iso(recording.requested_at), "format": MP4}
self.started[request_id] = answer
while len(self.started) > KEEP_STARTED:
Expand All @@ -173,11 +177,9 @@ async def start(self, request_id: str, scope: str, window: Optional[str], fps: O
return answer

@staticmethod
async def _live(recording: Optional[_Recording]) -> None:
"""Wait up to FIRST_FRAME_S for ``recording``, one still running, to have its first frame."""
async def _live(recording: Optional[Video]) -> None:
if recording is not None:
with contextlib.suppress(TimeoutError):
await asyncio.wait_for(recording.live.wait(), FIRST_FRAME_S)
await recording.until_live()

async def end(self, recording_id: str, upload: Optional[dict], reason: str) -> dict:
if reason not in REASONS:
Expand Down Expand Up @@ -215,18 +217,26 @@ def _hints(self, features: Recording, scope: str, window: Optional[str], fps: Op
return RecordHints(scope, fps, max_edge, bool(cursor), features.max_segment_s, Path()), grant


class _Recording:
class Video:
"""One recording's video: frames or segments from ``source(hints)``, encoded as H.264 segments in
``hints.directory``, each uploaded under ``grant`` as it closes, at ``<prefix>/<index>.mp4``, ``prefix`` its id by
default; sealed by ``end``, which answers its manifest and, with ``manifest``, uploads it beside the segments.
``streams`` says the source streams rather than polls, ``cursor`` that its frames draw the pointer."""

def __init__(self, recording_id: str, hints: RecordHints, features: Recording, root: Path,
grant: Optional[WriteNamespaceGrant], runtime: ComputerRuntime) -> None:
grant: Optional[WriteNamespaceGrant], source: Callable[[RecordHints], AsyncIterator], *,
streams: bool, cursor: bool, prefix: Optional[str] = None, manifest: bool = True) -> None:
self.id = recording_id
self.hints = hints
self.features = features
self.root = root
self.runtime = runtime
self.source = source
self.prefix = prefix or recording_id
self.uploads_manifest = manifest
self.uploader = NamespaceUploader(grant) if grant is not None else None
self.requested_at = datetime.now(timezone.utc)
self.streams = runtime.has("record")
self.cursor = hints.cursor if self.streams else hints.cursor and runtime.has("pointer")
self.streams = streams
self.cursor = cursor
self.segments: list[_Segment] = []
self.encoder: Optional[_Encoder] = None
self.ended_by: Optional[str] = None
Expand All @@ -241,6 +251,16 @@ def __init__(self, recording_id: str, hints: RecordHints, features: Recording, r
self.executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix=f"computer-{recording_id}")
self._last_disk_check = 0.0

def begin(self) -> None:
"""Recording from now, until ``end`` or ``max_duration_s``."""
self.task = asyncio.create_task(self.run())
self.timer = asyncio.get_running_loop().call_later(self.features.max_duration_s, self.expire)

async def until_live(self) -> None:
"""Wait up to FIRST_FRAME_S for it, still running, to have its first frame."""
with contextlib.suppress(TimeoutError):
await asyncio.wait_for(self.live.wait(), FIRST_FRAME_S)

async def run(self) -> None:
"""Frames or segments from the source, recorded until the recording ends; ``live`` once it's recording: a
stream from its start, the fallback's polling from its first frame, which a busy computer is slow to take."""
Expand Down Expand Up @@ -289,9 +309,7 @@ async def _release(self) -> None:
self.executor.shutdown(wait=False)

def _source(self) -> AsyncIterator:
if self.streams:
return self.runtime.stream(self.hints)
return _poll(self.runtime, self.hints, self.cursor)
return self.source(self.hints)

async def end(self, reason: str, grant: Optional[WriteNamespaceGrant]) -> dict:
if self.timer is not None:
Expand All @@ -318,9 +336,9 @@ async def end(self, reason: str, grant: Optional[WriteNamespaceGrant]) -> dict:
source="stream" if self.streams else "capture", started_at=entries[0].started_at if entries else None,
ended_at=entries[-1].ended_at if entries else None, ended_by=self.ended_by, segments=entries,
error=self.error if self.ended_by == "error" else None).wire()
if uploader is not None: # beside the segments, so a reader who didn't end the recording finds it
with contextlib.suppress(Exception):
await uploader.upload(f"{self.id}/{MANIFEST}", json.dumps(manifest).encode())
if uploader is not None and self.uploads_manifest: # beside the segments, so a reader who didn't end the
with contextlib.suppress(Exception): # recording finds it
await uploader.upload(f"{self.prefix}/{MANIFEST}", json.dumps(manifest).encode())
return manifest

async def _add_frame(self, item: RecordedFrame) -> None:
Expand Down Expand Up @@ -373,7 +391,7 @@ async def _finish(self, path: Path, started_at: datetime, ended_at: datetime, wi
frames: int) -> None:
size, digest = await asyncio.to_thread(_digest, path)
segment = _Segment(ManifestSegment(
index=len(self.segments), path=f"{self.id}/{path.name}", started_at=_iso(started_at),
index=len(self.segments), path=f"{self.prefix}/{path.name}", started_at=_iso(started_at),
ended_at=_iso(ended_at), width=width, height=height, frames=frames, bytes=size, sha256=digest,
uploaded=False), path)
self.segments.append(segment)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -472,7 +472,8 @@ def _answered(self, connection: _Connection, method: str, params: dict, result:
if self.contexts is None:
return
if method == MAKES_CONTEXT and isinstance(made := result.get("browserContextId"), str):
self.contexts.made_by_agent(connection.user, made)
self.contexts.made_by_agent(connection.user, made,
connection.scope.lease if connection.scope is not None else None)
if params.get("disposeOnDetach"):
connection.ends_on_detach.add(made)
elif method == ENDS_CONTEXT:
Expand Down Expand Up @@ -501,7 +502,8 @@ def _begin(self, connection: _Connection, session: str, method: str, params: dic
url = self.control.url_of(target) if self.control is not None and target else None
crediting = [call for call in self.calls.values() if call.user == connection.user]
action = {"via": "cdp", "who": connection.user, "connection": connection.number, "method": method,
"acts": acting, "tab": target, "url": url}
"acts": acting, "tab": target, "url": url,
**({"context": connection.record["context"]} if "context" in connection.record else {})}
if crediting:
action["for"] = crediting[0].begun["seq"] if len(crediting) == 1 else [c.begun["seq"] for c in crediting]
return _Logged(self, self.log.begin(action), method, params, target, acting, crediting)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,10 @@ def __init__(self, devtools: DevToolsPipe, control: BrowserControl,
home: Callable[[], Path] = Path.home) -> None:
self.devtools, self.control, self.home = devtools, control, home
self.leases: dict[str, Lease] = {}
#: What hears of each lease's end, whatever ends it: its release, its disposal or the browser's.
#: What hears of each lease's end, whatever ends it: its release, its disposal or the browser's; and of each
#: lease made, with the context the connection that made it is scoped to, None for the default context's.
self.on_end: list[Callable[[Lease], None]] = []
self.on_made: list[Callable[[Lease, Optional[str]], None]] = []
self._made: dict[str, int] = {}
self._tasks: set[asyncio.Future] = set()
self._lock = asyncio.Lock()
Expand Down Expand Up @@ -143,12 +145,13 @@ async def release(self, label: str) -> bool:
await asyncio.to_thread(shutil.rmtree, lease.folder(self.home()), True)
return True

def made_by_agent(self, user: str, context_id: str) -> Lease:
"""The context ``context_id`` an agent made over CDP as ``user``, leased to it as ``agent-<user>-<n>``,
starting empty, its downloads sent to its own folder."""
def made_by_agent(self, user: str, context_id: str, scope: Optional[str] = None) -> Lease:
"""The context ``context_id`` an agent made over CDP as ``user``, on a connection scoped to the context
``scope``, None for the default one, leased to it as ``agent-<user>-<n>``, starting empty, its downloads sent
to its own folder."""
if (known := self.of(context_id)) is not None:
return known
lease = self._record(Lease(self._agent_label(user), context_id, user, [user]))
lease = self._record(Lease(self._agent_label(user), context_id, user, [user]), scope)
self._spawn(self._send_downloads(lease))
return lease

Expand Down Expand Up @@ -219,10 +222,15 @@ def _ended(self, lease: Lease) -> None:
except Exception as e:
logger.warning(f"the end of the lease {lease.label!r}: {type(e).__name__}: {e}")

def _record(self, lease: Lease) -> Lease:
def _record(self, lease: Lease, scope: Optional[str] = None) -> Lease:
self.leases[lease.label] = lease
self.control.other_contexts[lease.id] = lease.label
lease.downloads(self.home()).mkdir(parents=True, exist_ok=True)
for hook in self.on_made:
try:
hook(lease, scope)
except Exception as e:
logger.warning(f"the lease {lease.label!r} made: {type(e).__name__}: {e}")
return lease

def _agent_label(self, user: str) -> str:
Expand Down
Loading
Loading