Skip to content

Commit b7a8e54

Browse files
feat(browser): start tabs load in order and settle before agents get an address (FD-4941) (#530)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 9efeead commit b7a8e54

20 files changed

Lines changed: 1108 additions & 103 deletions

File tree

‎packages/agentenv-protocol/README.md‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -880,14 +880,14 @@ under the session user's home (`.config/chromium` by default), whose contents fr
880880
are the profile's template. The base class adds:
881881

882882
- **Its settings**, `BrowserSettings`, which a subclass's `Settings` extends: `start_urls`, opened in
883-
tabs at launch, the first active (`["about:blank"]` by default; none may start with `-`), headless
884-
the first from its command line and the rest over its pipe, since headless Chromium takes one; `window`,
883+
tabs at launch, the first active (`["about:blank"]` by default; none may start with `-`); `window`,
885884
1280 × 800, alone only; `locale`; `time_zone`, set as `TZ` in its launch's `env`; `geolocation` and
886885
`notifications`, `deny` (the default), which answers a page's prompt for it, or `ask`; and `dialogs`, who
887886
answers a page's JavaScript dialogs: `agent` (the default), or agentenv, accepting (`accept`, a `prompt()`
888887
with its default text) or dismissing (`dismiss`) each. Either way agentenv leaves a page that asks before
889888
it's left (`beforeunload`), so none of its answers blocks a navigation.
890-
- **The flags agentenv owns**, after the subclass's command and before the start tabs: its profile, its
889+
- **The flags agentenv owns**, after the subclass's command and before `about:blank`, the one tab its command line
890+
opens: its profile, its
891891
HTTP cache under `~/.cache` (`--disk-cache-dir`), `--remote-debugging-pipe`, `--no-first-run`,
892892
`--no-default-browser-check`, `--lang` and, when `/dev/shm` holds under 1 GiB (Docker gives 64 MiB),
893893
`--disable-dev-shm-usage`, so it keeps its shared memory in files. With
@@ -946,6 +946,8 @@ are the profile's template. The base class adds:
946946
- `files/choose {paths}`, files in `~/Uploads` for the file chooser open on it, which headless, and so
947947
alone, it intercepts;
948948
- `page/state`, the active tab's id, URL and title and the dialog or file chooser open on it.
949+
- `sites/set {sites}`, the env's own sites, each `{name, domains}`, a `*.` domain matching one label more, which
950+
the deploy hands it before it launches the browser and its server keeps for every launch after.
949951
- `cookies/get {urls?}`, `cookies/set {cookies}` and `cookies/delete {name?, domain?, path?}`, on its default
950952
context, each cookie as Playwright's `storageState` has it, CDP's `Network.Cookie` fields (`name`, `value`,
951953
`domain`, `path`, `expires`, `httpOnly`, `secure`, `sameSite`), so a storage state exported from Playwright loads
@@ -968,9 +970,26 @@ are the profile's template. The base class adds:
968970
under the `write_namespace` grant when there is one, else inline, as `data/get` returns files. Inline, the whole
969971
file is read into memory and sent base64 in one answer, so a large one belongs under a grant.
970972

973+
**Start tabs.** At each launch its start tabs open over its pipe, in a computer and alone alike: the first in the
974+
blank tab its command line opened, the rest in tabs it opens blank after it, each held to the window's size
975+
(headless) before it goes to its URL, so its page sees that size from its first script; then the first is brought
976+
to the front. They load at once, and each settles once it has loaded (its load event fired, its document's final
977+
status read), once it failed (a network error, or on one of the env's sites, as `sites/set` named them, a 4xx or
978+
5xx status, which settles it at once), or once it has been waited on: every one settles within the browser's
979+
ready timeout of its launch, the launch's setup counted against it, a tab on one of the env's sites waited on what's
980+
left of it and any other on 30 s at most, so `data/reset`, a load and a deploy, which wait that timeout and a margin,
981+
always get its outcome. Its `start_tabs_loaded` ready check passes once all have settled, so `cdp` and a deploy wait
982+
for them. `record`'s `start_tabs` keeps each launch's (`StartTabsLaunch`: `at` and its
983+
`tabs`, each a `StartTab`: `tab`, `url`, `final_url`, `status`, `error`, `load_ms`, `site` — the env's site's
984+
name, null for a live one — and `outcome`, `loaded`, `failed` or `timed_out`), and `data/get`'s state keeps its
985+
latest launch's. A deploy of a computer reads them once it's ready, keeps them in its record and fails, naming
986+
each, if one on the env's sites didn't load; `data/reset` and a load that opens the start tabs again fail
987+
the same way, while a server's restart only records it.
988+
971989
The browser's data plane is a file tree, every file by its path under the session user's home:
972990
- `data/get` returns its state as `browser-state.json` (`BrowserState`: `cookies` and `local_storage`
973-
as above, `tabs`, `closed_tabs`, `downloads` and `clock`, `real` with its `now`); its profile's files,
991+
as above, `tabs`, `closed_tabs`, `downloads`, `clock`, `real` with its `now`, and `start_tabs`); its
992+
profile's files,
974993
each database with its journal or log, copied in one pass with the browser paused once its state is
975994
read, so the files are of one moment, without caches (`Cache`, `Code Cache`,
976995
`GPUCache`, the shader and Dawn caches, `CacheStorage`, `ScriptCache`, `Crashpad`) or locks

‎packages/agentenv-protocol/src/agentenv_protocol/computer/app.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -446,7 +446,7 @@ def __init__(self) -> None:
446446
self._computer_app_data = AppDataPlane(
447447
self._computer_app, {op: getattr(self, name) for op, name in cls._data_hooks.items()}, self.paths,
448448
cls.data_timeout_s, live=cls.data_live, keeps_paths=cls.data_keeps_paths,
449-
resets_failed_add=self._resets_failed_add)
449+
resets_failed_add=self._resets_failed_add, relaunched=self._relaunched)
450450

451451
def _launch_for(self, launch: Launch) -> Launch:
452452
"""The launch its server uses, from the one declared for its OS, whose ``ready`` timeout it keeps: as
@@ -473,6 +473,9 @@ def _resets_failed_add(self) -> bool:
473473
"""Whether a load that fails is followed by a reset, so the app isn't left half loaded: always."""
474474
return True
475475

476+
async def _relaunched(self, op: str) -> None:
477+
"""What a data call that stopped the app, ``op``, awaits once it has launched it again: nothing."""
478+
476479
@classmethod
477480
def settings_on(cls, os_type: Optional[str]) -> type[BaseModel]:
478481
"""Its ``Settings`` on ``os_type``, with the fields its OS adds."""

‎packages/agentenv-protocol/src/agentenv_protocol/computer/app_data.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
import logging
1212
import shutil
1313
import tempfile
14-
from collections.abc import Callable
14+
from collections.abc import Awaitable, Callable
1515
from pathlib import Path, PurePosixPath
1616
from typing import TYPE_CHECKING, Any, Optional
1717

@@ -41,14 +41,17 @@ class AppDataPlane:
4141

4242
def __init__(self, runtime: AppRuntime, hooks: dict[str, Callable], paths: list[str], timeout_s: float, *,
4343
live: frozenset[str] = frozenset(), keeps_paths: bool = False,
44-
resets_failed_add: Callable[[], bool] = lambda: True) -> None:
44+
resets_failed_add: Callable[[], bool] = lambda: True,
45+
relaunched: Optional[Callable[[str], Awaitable[None]]] = None) -> None:
4546
self.runtime = runtime
4647
self.hooks = hooks
4748
self.paths = paths
4849
self.timeout_s = timeout_s
4950
#: The operations served on the running app; whether a hook's reset needs the paths as installed; and whether
5051
#: a load that fails is followed by a reset now.
5152
self.live, self.keeps_paths, self.resets_failed_add = live, keeps_paths, resets_failed_add
53+
#: What a call that stopped the app awaits once it has launched it again, whose error fails the call.
54+
self.relaunched = relaunched
5255
self.turn = asyncio.Lock()
5356
self.changing: set[asyncio.Task] = set()
5457

@@ -99,6 +102,8 @@ async def _change(self, op: str, served: Callable, args: tuple, answer: asyncio.
99102
finally:
100103
if running:
101104
await self.runtime.launch_app(_CAUSES[op])
105+
if running and self.relaunched is not None:
106+
await self.relaunched(op)
102107
except Exception as e:
103108
if not answer.done():
104109
answer.set_exception(e)

‎packages/agentenv-protocol/src/agentenv_protocol/website_browser/__init__.py‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -5,19 +5,21 @@
55
skill and ``browser/v1`` on its card. It runs alone, headless, on a computer with no display, a browser computer's,
66
whose screen is its page; which site proxy it goes through comes from a variable agentenv sets, never from its
77
settings."""
8-
from .control import (ACTIVE_CHECK_TIMEOUT_S, AGENT, AGENTENV, CAPTURE_TIMEOUT_S, DOWNLOADS, EXIT, HASH_CHUNK_BYTES,
9-
PAGE_INPUT, PAGES, PIN_TIMEOUT_S, STOP, UNKNOWN, UNNAVIGATED_CAPTURE_TIMEOUT_S, UPLOADS,
10-
BrowserControl)
8+
from .control import (ACTIVE_CHECK_TIMEOUT_S, AGENT, AGENTENV, CAPTURE_TIMEOUT_S, DOWNLOADS, EXIT, FAILED,
9+
HASH_CHUNK_BYTES, LIVE_START_TAB_TIMEOUT_S, LOADED, PAGE_INPUT, PAGES, PIN_TIMEOUT_S, STOP,
10+
TIMED_OUT, UNKNOWN, UNNAVIGATED_CAPTURE_TIMEOUT_S, UPLOADS, BrowserControl)
1111
from .devtools import MAX_MESSAGE_BYTES, PIPE_IN, PIPE_OUT, DevToolsError, DevToolsPipe
1212
from .environment import (ALONE_BYPASS, BLOCK, BROWSER, COMPUTER_BYPASS, OWNED_FLAGS, POLICIES, READY_POLL_S,
13-
SHM_MIN_BYTES, SITE_PROXY_ENV, SKILL, BrowserSettings, WebsiteBrowserEnvironment, Window)
13+
SHM_MIN_BYTES, SITE_PROXY_ENV, SITES, SKILL, BrowserSettings, WebsiteBrowserEnvironment,
14+
Window, start_tab_why)
1415
from .shapes import (BrowserRecord, ClosedTab, Cookie, CookieToSet, Dialog, Download, LaunchEntry, OpenTab,
15-
OriginStorage, PartitionKey, StorageItem, origin_of)
16+
OriginStorage, PartitionKey, SiteNames, StartTab, StartTabsLaunch, StorageItem, origin_of)
1617

1718
__all__ = ["ACTIVE_CHECK_TIMEOUT_S", "AGENT", "AGENTENV", "ALONE_BYPASS", "BLOCK", "BROWSER", "CAPTURE_TIMEOUT_S",
18-
"COMPUTER_BYPASS", "DOWNLOADS", "EXIT", "HASH_CHUNK_BYTES", "MAX_MESSAGE_BYTES", "OWNED_FLAGS", "PAGE_INPUT",
19-
"PAGES", "PIN_TIMEOUT_S", "PIPE_IN", "PIPE_OUT", "POLICIES", "READY_POLL_S", "SHM_MIN_BYTES",
20-
"SITE_PROXY_ENV", "SKILL", "STOP", "UNKNOWN", "UNNAVIGATED_CAPTURE_TIMEOUT_S", "UPLOADS", "BrowserControl",
21-
"BrowserSettings", "DevToolsError", "DevToolsPipe", "WebsiteBrowserEnvironment", "Window",
22-
"BrowserRecord", "ClosedTab", "Cookie", "CookieToSet", "Dialog", "Download", "LaunchEntry", "OpenTab",
23-
"OriginStorage", "PartitionKey", "StorageItem", "origin_of"]
19+
"COMPUTER_BYPASS", "DOWNLOADS", "EXIT", "FAILED", "HASH_CHUNK_BYTES", "LIVE_START_TAB_TIMEOUT_S", "LOADED",
20+
"MAX_MESSAGE_BYTES", "OWNED_FLAGS", "PAGE_INPUT", "PAGES", "PIN_TIMEOUT_S", "PIPE_IN", "PIPE_OUT",
21+
"POLICIES", "READY_POLL_S", "SHM_MIN_BYTES", "SITE_PROXY_ENV", "SITES", "SKILL", "STOP", "TIMED_OUT",
22+
"UNKNOWN", "UNNAVIGATED_CAPTURE_TIMEOUT_S", "UPLOADS", "BrowserControl", "BrowserSettings", "DevToolsError",
23+
"DevToolsPipe", "WebsiteBrowserEnvironment", "Window", "BrowserRecord", "ClosedTab", "Cookie", "CookieToSet",
24+
"Dialog", "Download", "LaunchEntry", "OpenTab", "OriginStorage", "PartitionKey", "SiteNames", "StartTab",
25+
"StartTabsLaunch", "StorageItem", "origin_of", "start_tab_why"]

0 commit comments

Comments
 (0)