Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions src/agent_env/computers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,12 @@ building it. Name it from another bundle as `ubuntu-24.04`, the alias `aliases.t
`artifacts/ubuntu-24.04-headless/` is Ubuntu 24.04 (`amd64` and `arm64`) with no display server or desktop
(`display = "none"`), for one website browser, which runs headless there, and the `page` driver, whose screen is the
browser's page: a browser computer, which agentenv's `chromium` deploys alone as. supervisord runs every program as the
session user, among them a stand-in site proxy on `127.0.0.1:3128`, the browser's only route out, which forwards every
request to wherever it names, refusing nothing: the sandbox's network policy is all that limits it until the shared
site proxy replaces it. It has a container form only. Name it from another bundle as `ubuntu-24.04-headless`, or by
its id, `@local/agentenv-framework/computers/ubuntu-24.04-headless`.
session user, among them a stand-in site proxy on `127.0.0.1:3128`, the browser's only route out when the computer
routes no sites, which forwards every request to wherever it names, refusing nothing: the sandbox's network policy is
all that limits it. A computer that routes sites names the shared site proxy's forward port, `127.0.0.1:3129`, to the
browser instead: it serves the run's sites, refuses every other host and runs out of the agent's reach. It has a
container form only. Name it from another bundle as `ubuntu-24.04-headless`, or by its id,
`@local/agentenv-framework/computers/ubuntu-24.04-headless`.

It also has a disk form, a disk a VM boots from UEFI, which is the container form's root filesystem on the Linux boot
recipe in `disk/linux/`: a GPT disk of an EFI system partition holding one UKI, Ubuntu's kernel with a busybox initramfs,
Expand Down
104 changes: 80 additions & 24 deletions src/agent_env/env/envs/computer/utils/site_proxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,19 @@
``python3``, so it uses the standard library alone.

``install``, as root, maps each exact name to loopback in the hosts file, trusts the deployment's CA in the system
store, Firefox's policies, Chromium's managed policies and the session user's NSS databases, starts ``serve`` and checks, as the session user, that
each name loads. ``serve`` listens on loopback: on 443 for a name served over HTTPS, on 80 to redirect those to HTTPS
and to serve ``.localhost`` names over HTTP, and on 53 for DNS, over UDP and TCP, when a name is a wildcard, which a
hosts file can't list; it then drops to ``SERVE_USER`` (``nobody``), else the session user, so the agent, which runs as
the session user, can't stop it, and ``keep``, as root, starts it again should it end. The site sees itself at ``/`` under its own name: the path goes upstream under
the route's prefix, ``X-Forwarded-Host`` and ``X-Forwarded-Proto`` say the name and scheme the browser used,
``AgentEnv-Site-Root`` has the gateway tell the site no prefix, so the links it makes are rooted at ``/`` too, and a
``Location`` or cookie naming the route is rewritten back to the name. Hop-by-hop headers and the browser's own
``X-Forwarded-*`` never go upstream; ``Transfer-Encoding`` does, since bodies are relayed as framed, and an
``Alt-Svc`` offering HTTP/3, which the proxy doesn't serve, never reaches the browser. A WebSocket's upgrade is
relayed, for an upstream that answers it."""
store, Firefox's policies, Chromium's managed policies and the session user's NSS databases, starts ``serve`` and
checks, as the session user, that each name loads. ``serve`` listens on loopback: on 443 for a name served over HTTPS,
on 80 to redirect those to HTTPS and to serve ``.localhost`` names over HTTP, on 53 for DNS, over UDP and TCP, when a
name is a wildcard, which a hosts file can't list, and, on a browser computer, on ``proxy_port`` (3129) as a forward
proxy, a browser alone's only route out, which serves the same sites to ``CONNECT`` and to requests naming their URLs
and refuses any other host at once; it then drops to ``SERVE_USER`` (``nobody``), else the session user, so the agent,
which runs as the session user, can't stop it, and ``keep``, as root, starts it again should it end. The site sees
itself at ``/`` under its own name: the path goes upstream under the route's prefix, ``X-Forwarded-Host`` and
``X-Forwarded-Proto`` say the name and scheme the browser used, ``AgentEnv-Site-Root`` has the gateway tell the site no
prefix, so the links it makes are rooted at ``/`` too, and a ``Location`` or cookie naming the route is rewritten back
to the name. Hop-by-hop headers and the browser's own ``X-Forwarded-*`` never go upstream; ``Transfer-Encoding`` does,
since bodies are relayed as framed, and an ``Alt-Svc`` offering HTTP/3, which the proxy doesn't serve, never reaches the
browser. A WebSocket's upgrade is relayed, for an upstream that answers it."""

from __future__ import annotations

Expand Down Expand Up @@ -143,6 +145,7 @@ class Config:
listen: str = LOOPBACK
https_port: int = 443
http_port: int = 80
proxy_port: Optional[int] = None
dns_port: Optional[int] = None
nameservers: list[str] = field(default_factory=list)

Expand All @@ -162,7 +165,8 @@ def load(cls, path: str) -> Config:
return cls(user=data.get("user"), sites=sites, serve_user=data.get("serve_user"),
listen=data.get("listen", LOOPBACK),
https_port=data.get("https_port", 443), http_port=data.get("http_port", 80),
dns_port=data.get("dns_port"), nameservers=list(data.get("nameservers") or []))
proxy_port=data.get("proxy_port"), dns_port=data.get("dns_port"),
nameservers=list(data.get("nameservers") or []))

def site_for(self, host: str) -> Optional[Site]:
"""The site ``host`` names: one naming it exactly before one whose wildcard covers it."""
Expand Down Expand Up @@ -361,15 +365,61 @@ class SiteProxy:
"""The proxy's HTTP side: one connection's requests, each forwarded on a connection of its own to its site's
route."""

def __init__(self, config: Config):
def __init__(self, config: Config, tls: Optional[ssl.SSLContext] = None):
self.config = config
self.tls = tls
self.upstream_tls = ssl.create_default_context()

async def handle(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter, scheme: str) -> None:
await self._serving(reader, writer, scheme, self._requests)

async def handle_forward(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
"""The forward-proxy port's connection: a ``CONNECT`` to a site's name tunnels to the site as the listeners
serve it, over TLS with its certificate for a name served over HTTPS; a request naming a site's URL is served
as the HTTP listener serves it; any other host, loopback among them, is refused at once and logged."""
await self._serving(reader, writer, "proxy", self._proxied)

async def _requests(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter, scheme: str,
peer: str) -> None:
while await self.exchange(reader, writer, scheme, peer):
pass

async def _proxied(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter, _: str, peer: str) -> None:
while (head := await read_head(reader)) is not None:
start, headers = parse_head(head)
method, _, target = start.partition(" ")
target = target.rpartition(" ")[0]
if method == "CONNECT":
host = split_host(target)
if (site := self.config.site_for(host)) is None:
self._refuse(writer, method, target)
return
writer.write(b"HTTP/1.1 200 Connection established\r\n\r\n")
await writer.drain()
tls = site.tls(host.lower())
if tls:
if self.tls is None:
raise BadRequest(f"{host} is served over HTTPS, and this proxy has no certificate for it")
await writer.start_tls(self.tls)
await self._requests(reader, writer, "https" if tls else "http", peer)
return
named = urlsplit(target).netloc if "://" in target else (header(headers, "host") or "")
if self.config.site_for(split_host(named)) is None:
self._refuse(writer, method, named or target)
return
if not await self.exchange(reader, writer, "http", peer, head=head):
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Resolved
return

@staticmethod
def _refuse(writer: asyncio.StreamWriter, method: str, host: str) -> None:
print(f"site proxy: refused {method} {host}: no site of this computer", file=sys.stderr, flush=True)
writer.write(simple_response("403 Forbidden", f"{host} is no site of this computer, and the browser reaches "
"its sites alone\n", FAILED))

async def _serving(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter, scheme: str, run) -> None:
peer = (writer.get_extra_info("peername") or (LOOPBACK,))[0]
try:
while await self.exchange(reader, writer, scheme, peer):
pass
await run(reader, writer, scheme, peer)
except BadRequest as e:
writer.write(simple_response("400 Bad Request", f"{e}\n"))
except (ConnectionError, asyncio.IncompleteReadError, ssl.SSLError, OSError):
Expand All @@ -385,24 +435,26 @@ async def handle(self, reader: asyncio.StreamReader, writer: asyncio.StreamWrite
pass

async def exchange(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter, scheme: str,
peer: str) -> bool:
"""One request and its answer; whether the connection takes another."""
head = await read_head(reader)
peer: str, head: Optional[bytes] = None) -> bool:
"""One request, read unless its ``head`` is given, and its answer; whether the connection takes another."""
head = head or await read_head(reader)
if head is None:
return False
start, headers = parse_head(head)
try:
method, target, version = start.split(" ")
except ValueError:
raise BadRequest(f"a malformed request line {start!r}") from None
host = header(headers, "host") or (urlsplit(target).netloc if "://" in target else "")
host = (urlsplit(target).netloc if "://" in target else "") or header(headers, "host") or ""
site = self.config.site_for(split_host(host)) if host else None
if site is None:
writer.write(simple_response("421 Misdirected Request", f"{host or 'no host'} is no site of this "
"computer\n", FAILED))
return False
if scheme == "http" and site.tls(split_host(host).lower()):
location = f"https://{split_host(host)}{target if target.startswith('/') else urlsplit(target).path}"
parts = urlsplit(target)
path = target if target.startswith("/") else urlunsplit(("", "", parts.path or "/", parts.query, ""))
location = f"https://{split_host(host)}{path}"
writer.write(simple_response("308 Permanent Redirect", f"{location}\n", (("Location", location),)))
return False
upgrade = header(headers, "upgrade") if "upgrade" in tokens(headers, "connection") else None
Expand Down Expand Up @@ -602,13 +654,17 @@ def pick(connection: ssl.SSLObject, name: Optional[str], _: ssl.SSLContext) -> N

async def serve(config: Config) -> None:
"""Listen, drop to the user it serves as, and serve until cancelled."""
proxy = SiteProxy(config)
context = tls_context(config)
proxy = SiteProxy(config, tls=context)
servers = []
if (context := tls_context(config)) is not None:
if context is not None:
servers.append(await asyncio.start_server(lambda r, w: proxy.handle(r, w, "https"), config.listen,
config.https_port, ssl=context, limit=HEAD_LIMIT))
servers.append(await asyncio.start_server(lambda r, w: proxy.handle(r, w, "http"), config.listen,
config.http_port, limit=HEAD_LIMIT))
if config.proxy_port:
servers.append(await asyncio.start_server(proxy.handle_forward, config.listen, config.proxy_port,
limit=HEAD_LIMIT))
transport = None
if config.dns_port:
resolver = Resolver(config)
Expand Down Expand Up @@ -932,8 +988,8 @@ def install(config_path: str, paths: Paths = Paths()) -> dict:
write_in_place(paths.profile, "".join(f"export {name}={value}\n"
for name, value in given["trust_environment"].items()))
trust["environment"] = str(paths.profile)
serving = {"user": user, "serve_user": serve_user(user), "sites": sites, **{key: given[key] for key in ("listen", "https_port", "http_port")
if key in given}}
serving = {"user": user, "serve_user": serve_user(user), "sites": sites,
**{key: given[key] for key in ("listen", "https_port", "http_port", "proxy_port") if key in given}}
if wildcard:
serving.update(dns_port=given.get("dns_port", 53), nameservers=nameservers)
if upstream_headers:
Expand Down
28 changes: 22 additions & 6 deletions src/agent_env/env/envs/computer/utils/sites.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
import httpx
from agentenv_protocol import client as protocol_client
from agentenv_protocol.website import WEBSITE_EXTENSION_URI as WEBSITE
from agentenv_protocol.website_browser import SITE_PROXY_ENV
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec
Expand All @@ -55,6 +56,11 @@
#: By the OS type of each computer site routing reaches, the variables naming the trust files agentenv adds, which a
#: computer routing sites boots with.
TRUST_ENVIRONMENTS = {"linux": dict(LOCAL_TRUST_ENV)}
#: The site proxy's forward port, a browser alone's only route out when its computer routes sites (the base's own
#: stand-in, for a computer that routes none, keeps 3128), and the display of the computer a browser runs alone on.
FORWARD_PROXY_PORT = 3129
FORWARD_PROXY = f"http://127.0.0.1:{FORWARD_PROXY_PORT}"
ALONE_DISPLAY = "none"
LOG = FOLDER / "site-proxy.log"
SYNTHETIC, ALLOWLIST = "synthetic", "allowlist"
RANKS = (SYNTHETIC, ALLOWLIST)
Expand Down Expand Up @@ -284,12 +290,19 @@ def trust_environment(os_type: str, sites: list[Site]) -> dict[str, str]:
return dict(TRUST_ENVIRONMENTS[os_type]) if any(route.tls_names for route in site_table(sites).routes) else {}


def proxy_config(table: SiteTable, issued: Issued, *, user: Optional[str], os_type: str,
def browser_environment(display: str) -> dict[str, str]:
"""For a computer routing sites, the variable naming the forward port to a browser alone, which runs on a computer
with no display; none on one with a display, whose apps reach the sites by name on the transparent listeners."""
return {SITE_PROXY_ENV: FORWARD_PROXY} if display == ALONE_DISPLAY else {}


def proxy_config(table: SiteTable, issued: Issued, *, user: Optional[str], os_type: str, forward_port: bool,
upstream_headers: Optional[dict[str, dict[str, str]]] = None) -> dict:
"""What ``site_proxy.py install`` reads: the session user, the CA, the trust files on an ``os_type`` computer and
the variables naming them, which its profile script sets as the computer's boot environment does, each route with
its certificate and key, and the headers its gateway's sandbox provider asks requests to it to carry
(``upstream_headers``, by site), which the proxy sends upstream alone."""
(``upstream_headers``, by site), which the proxy sends upstream alone, and with ``forward_port``, for a browser
alone, the forward port."""
sites = []
for route in table.routes:
site = {"name": route.site, "upstream": route.url, "domains": list(route.domains)}
Expand All @@ -299,7 +312,8 @@ def proxy_config(table: SiteTable, issued: Issued, *, user: Optional[str], os_ty
site["upstream_headers"] = dict(upstream_headers[route.site])
sites.append(site)
return {"user": user, "ca_pem": issued.ca_pem, "sites": sites, "log": str(LOG),
"trust_files": trust_files(os_type), "trust_environment": dict(TRUST_ENVIRONMENTS[os_type])}
"trust_files": trust_files(os_type), "trust_environment": dict(TRUST_ENVIRONMENTS[os_type]),
**({"proxy_port": FORWARD_PROXY_PORT} if forward_port else {})}


def reachable_problem(policy: Optional[NetworkPolicy], routes: list[Route]) -> Optional[str]:
Expand Down Expand Up @@ -404,10 +418,11 @@ async def _exec(self, argv: list[str]) -> str:


async def route(sandbox: Sandbox, sites: list[Site], *, user: Optional[str], network: Optional[dict],
valid_for: timedelta, label: str, os_type: str) -> dict:
valid_for: timedelta, label: str, os_type: str, display: str) -> dict:
"""Have the ``os_type`` computer on ``sandbox`` reach each of ``sites`` by its names: the table made, the CA and
certificates issued, the proxy installed and started as root, and each name checked as ``user``; what the deploy
record keeps of it. Raises naming a name that didn't load."""
certificates issued, the proxy installed and started as root, with its forward port on a computer with no
``display``, a browser alone's, and each name checked as ``user``; what the deploy record keeps of it. Raises
naming a name that didn't load."""
requested = NetworkPolicy.from_dict(network) if network else None
table = site_table(sites, requested.allow_hosts if requested else ())
table.routes = [Route(r.site, sandbox.url_from_sandbox(r.url), r.domains) for r in table.routes]
Expand All @@ -424,6 +439,7 @@ async def route(sandbox: Sandbox, sites: list[Site], *, user: Optional[str], net
await channel.private_folder(FOLDER)
await channel.write(PROXY, _PROXY_SOURCE.read_text())
await channel.write_secret(CONFIG, json.dumps(proxy_config(table, issued, user=user, os_type=os_type,
forward_port=display == ALONE_DISPLAY,
upstream_headers=upstream_headers)))
await channel.run(["chmod", "0600", str(CONFIG)])
output = await channel.run(["python3", str(PROXY), "install", str(CONFIG)])
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ async def _route_sites(env: ComputerEnv, sandbox: Sandbox, params: computer_para
record keeps of it."""
routed = await sites.route(sandbox, params.sites, user=env.computer.session.user, network=env.network,
valid_for=timedelta(seconds=valid_for_s), label=env.environment_name,
os_type=env.computer.os.type)
os_type=env.computer.os.type, display=env.computer.os.display)
return {**routed, "valid_for_s": valid_for_s}


Expand Down Expand Up @@ -328,8 +328,8 @@ def _environment(env: ComputerEnv, computer: ComputerArtifact, values: dict,
driver_settings=params.driver_settings, tool_prefix=tool_prefix),
**app_settings_environment(env.composed_apps(), computer_params.app_settings(env, params.app_settings),
env.computer.os.type),
**({HOLD_BOOT_LAUNCH_ENV: "1", **sites.trust_environment(env.computer.os.type, params.sites)}
if params.site_routing else {})}
**({HOLD_BOOT_LAUNCH_ENV: "1", **sites.trust_environment(env.computer.os.type, params.sites),
**sites.browser_environment(env.computer.os.display)} if params.site_routing else {})}


def _require_name(env: ComputerEnv, card: dict) -> None:
Expand Down
Loading
Loading