Skip to content

Commit b0900a7

Browse files
feat(computer): boot Windows computers on modal_vm (#534)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent d1f9079 commit b0900a7

8 files changed

Lines changed: 295 additions & 78 deletions

File tree

‎src/agent_env/computers/README.md‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -38,10 +38,12 @@ before reset in place started is refused a reset, to be redeployed. It boots on
3838
Docker can pass it `/dev/kvm`, and on `modal_vm`, amd64 only, whose sandbox runs the launcher as its image under nested
3939
virtualization: the provider exports the disk from the image store with this machine's Docker, compressed, once for each
4040
composed disk, into the per-user cache, and mounts it on the sandbox at create, which Modal uploads once for each
41-
content and the launcher expands onto the sandbox's disk. Such a sandbox is ended once it has been idle, with no
42-
connection open, for the provider's `idle_timeout`, 30 minutes unless its config sets another, as a running QEMU isn't
43-
activity. A gate 3 probe that ends with the connection to the driver dropped, refused or timed out runs again on a fresh
44-
boot, up to three in all, and the record keeps each attempt; one with a failed check is recorded as it is.
41+
content and the launcher expands onto the sandbox's disk; a thin disk, a Windows one, carries its base too, exported
42+
once for each base and expanded beside it there, so another computer on that base adds its layer alone. Such a sandbox
43+
is ended once it has been idle, with no connection open, for the provider's `idle_timeout`, 30 minutes unless its config
44+
sets another, as a running QEMU isn't activity. A gate 3 probe that ends with the connection to the driver dropped,
45+
refused or timed out runs again on a fresh boot, up to three in all, and the record keeps each attempt; one with a
46+
failed check is recorded as it is.
4547

4648
A put, `env build` and the release compose every form; a run composes only the forms its sandbox's deploys boot: on each
4749
link of a chain, the first form it boots natively, as a deploy picks, so local Docker composes containers, and a sandbox
@@ -170,10 +172,10 @@ pulls it. Gate 3 probes it at full. At the session user's sign-in,
170172
`agentenv-session`, run windowless by `pythonw`, reads the deploy's environment from the NoCloud seed, which the
171173
launcher attaches as a removable USB drive, since Windows mounts a FAT filesystem with no partition table on removable
172174
media alone, and starts each layer's services in that session, again whenever one exits. A Windows computer boots
173-
with 4 vCPUs and 8 GB, on local Docker with KVM; it has computer-exec and data/add, through the protocol's Windows
174-
defaults, so the probe runs its fixture there as on Linux, with Python one-liners for the commands it would give a
175-
shell. A Windows computer composes its driver alone for now, no app, and boots on no Modal VM sandbox yet, whose export
176-
carries a disk's top layer alone.
175+
with 4 vCPUs and 8 GB, on local Docker with KVM and on `modal_vm`; it has computer-exec and data/add, through the
176+
protocol's Windows defaults, so the probe runs its fixture there as on Linux, with Python one-liners for the commands it
177+
would give a shell. On `modal_vm` it runs in your own Modal workspace, under the evaluation's terms or a licence of your
178+
own; whether they allow it there is yours to check. A Windows computer composes its driver alone for now, no app.
177179

178180
This bundle has no tasks of its own: other bundles pull its computers by alias, and `agent-env run computers` has
179181
nothing to run.

‎src/agent_env/computers/disk/linux/launcher/launch‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
# and, under KVM, Hyper-V enlightenments.
99
# VM_DISK the disk, /disk/disk.img by default, which may be backed by another beside it; one ending .zst is
1010
# expanded onto this machine's disk first
11+
# VM_BASE the base of a thin VM_DISK ending .zst, compressed too, which a provider carries beside it: expanded
12+
# beside the expanded disk, as the base.img it's backed by
1113
# VM_ENV_<NAME> the computer's variable NAME, its value base64'd, for its NoCloud seed (CIDATA; see seed)
1214
# VM_PORTS guest TCP ports, each served on the same port here by socat, whose listen backlog takes a burst of
1315
# connections, in front of QEMU's forward on 127.0.0.1 at the port plus 10000, whose backlog is 1;
@@ -111,6 +113,9 @@ trap 'kill "$(cat "$control/qemu.pid" 2>/dev/null)" 2>/dev/null; exit 143' TERM
111113
if [[ "$disk" == *.zst ]]; then
112114
zstd -q -d -f "$disk" -o "$state/disk.img"
113115
disk=$state/disk.img
116+
if [ -n "${VM_BASE:-}" ]; then
117+
zstd -q -d -f "$VM_BASE" -o "$state/base.img"
118+
fi
114119
fi
115120

116121
boot() {

‎src/agent_env/providers/sandbox_providers/AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ with `exposed_ports=[port]` and returns the VM, and the deploy then loads and st
5757
| `create_container(...)` | inherited | containers that run an env or agent image. Override only when the platform boots images natively |
5858
| `reset_container(sandbox, *, image_name, port, env)` | inherited | a computer env's reset, which replaces the container `create_container` ran with a fresh one on the same sandbox and port. The default works on a VM backend; one whose sandbox is the container itself raises `NotImplementedError` |
5959
| `native_platforms()` | no | a computer env's deploy and gate 3, which boot a build only on a provider that runs its arch without emulation. The default is `linux/amd64`; local Docker returns this machine's |
60-
| `disk_image_platforms()` | no | the same for a computer's disk form: the platforms whose disks `create_vm` boots without emulation. The default is none; local Docker returns this machine's where it can pass `/dev/kvm` to a container, and `modal_vm` `linux/amd64`, booting the launcher as its sandbox's image under nested virtualization |
60+
| `disk_image_platforms()` | no | the same for a computer's disk form: the platforms whose disks `create_vm` boots without emulation. The default is none; local Docker returns this machine's where it can pass `/dev/kvm` to a container, and `modal_vm` `linux/amd64` and `windows/amd64`, booting the launcher as its sandbox's image under nested virtualization, a Windows disk's base carried beside it |
6161
| `vm_bundle_platforms()` | no | the same for a computer's vm_bundle form, a whole saved VM (`format = "lume"`): the platforms, as `<os>/<arch>`, whose VM bundles `create_vm` boots. The default is none; `lume` returns `macos/arm64` on an Apple Silicon Mac. A provider that names one implements the VM-bundle methods below too |
6262
| `reset_vm(sandbox, *, image, env)` | no | a computer env's reset of a disk `create_vm` booted: boot it afresh on the same sandbox and ports. The default raises `NotImplementedError`; local Docker and `modal_vm` run the launcher's `launch reset` in the sandbox hosting it, with `env` on its stdin |
6363
| `has_vm_bundle(image)`, `delete_vm_bundle(image)`, `open_builder(image, name, build)`, `keep_builder(builder)`, `ssh_password` | when `vm_bundle_platforms()` names a platform | a macOS computer's compose, and a deploy on a Mac that doesn't hold the env's VM bundle: `open_builder` boots a clone of the base `image`, named `name`, with the folder `build` (named `agentenv-build`) shared read-write, and its sandbox's `exec_with_output` runs over ssh as the session user, whose sudo reads `ssh_password` from that folder; `keep_builder` shuts it down from inside and keeps it, returning `local:<name>`; `has_vm_bundle` says whether this machine holds a `local:` VM bundle, and `delete_vm_bundle` deletes one, as a compose does one that failed its probe. Here `create_vm` boots `image`, `local:<name>` or a registry reference pinned by digest, with `env` as the protocol's seed in a folder the VM shares, and `reset_vm` clones it afresh. See `lume/provider.py` |

‎src/agent_env/providers/sandbox_providers/local_docker/disk.py‎

Lines changed: 129 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,10 @@
66
import asyncio
77
import hashlib
88
import os
9+
import shutil
910
import subprocess
1011
import tempfile
12+
from dataclasses import dataclass
1113
from pathlib import Path, PurePosixPath
1214

1315
from agent_env.artifact.artifacts.computer import DISK_BASE_FILE, DISK_FILE, DISK_GUEST_FILE
@@ -17,11 +19,13 @@
1719
from agent_env.env.published_images import LAUNCHER_IMAGE, published_pinned
1820
from agent_env.providers.sandbox_providers.disk_launcher import launcher_variables
1921
from agent_env.providers.sandbox_providers.host_images import host_platform, present
20-
from agent_env.providers.sandbox_providers.local_docker.cli import docker
22+
from agent_env.providers.sandbox_providers.local_docker.cli import DockerFailed, docker
2123
from agent_env.store.local_state import holding_locks
2224
from agent_env.utils.docker.build import build_image
2325

2426
KVM = "/dev/kvm"
27+
#: What ``docker cp`` says of a path its container lacks: Docker 23 and later, then earlier.
28+
_NO_SUCH_PATH = ("Could not find the file", "No such container:path")
2529

2630

2731
def kvm_platforms() -> frozenset[str]:
@@ -35,62 +39,153 @@ def kvm_platforms() -> frozenset[str]:
3539

3640
async def machine_image(disk: str) -> str:
3741
"""An image of the launcher holding ``disk``'s files for this machine's platform, built unless held: one for each
38-
image ``disk`` names, by its id, so a disk built again under its tag gets a machine of its own. A disk of more than
39-
one layer is a thin one on ``DISK_BASE_FILE``, which gets a step of its own, so every machine on that base shares
40-
one copy of it."""
42+
image ``disk`` names, by its id, so a disk built again under its tag gets a machine of its own. A thin disk's base
43+
gets a step of its own, so every machine on that base shares one copy of it."""
4144
source = await asyncio.to_thread(launcher)
42-
identity, layers = (await asyncio.to_thread(docker, "image", "inspect", "-f", "{{.Id}} {{len .RootFS.Layers}}",
43-
disk)).split()
44-
tag = "agentenv-vm:" + hashlib.sha256(f"{source} {disk} {identity}".encode()).hexdigest()[:16]
45+
layout = await asyncio.to_thread(_layout, disk)
46+
tag = "agentenv-vm:" + hashlib.sha256(f"{source} {disk} {layout.identity}".encode()).hexdigest()[:16]
4547
if not await asyncio.to_thread(present, tag):
46-
layers = int(layers)
47-
await asyncio.to_thread(_build, _machine_dockerfile(source, disk, layers), tag)
48+
await asyncio.to_thread(_build, _machine_dockerfile(source, disk, thin=layout.base is not None), tag)
4849
return tag
4950

5051

51-
def _machine_dockerfile(launcher: str, disk: str, layers: int) -> str:
52+
def _machine_dockerfile(launcher: str, disk: str, thin: bool) -> str:
5253
folder = f"/{PurePosixPath(DISK_FILE).parent}/"
53-
if layers == 1:
54+
if not thin:
5455
return f"FROM {launcher}\nCOPY --from={disk} {folder} {folder}\n"
5556
return (f"FROM {launcher}\nCOPY --from={disk} /{DISK_BASE_FILE} /{DISK_BASE_FILE}\n"
5657
f"COPY --from={disk} /{DISK_FILE} /{DISK_GUEST_FILE} {folder}\n")
5758

5859

60+
@dataclass(frozen=True)
61+
class _Layout:
62+
"""A disk image's id and, for a thin disk, one of more than one layer on ``DISK_BASE_FILE``, its ``base``: the
63+
digest of its first layer, the base's own, which every disk composed on that base shares; None for a whole one."""
64+
65+
identity: str
66+
base: str | None
67+
68+
69+
def _layout(disk: str) -> _Layout:
70+
identity, layers, first = docker("image", "inspect", "-f",
71+
"{{.Id}} {{len .RootFS.Layers}} {{index .RootFS.Layers 0}}", disk).split()
72+
return _Layout(identity, first if int(layers) > 1 else None)
73+
74+
5975
def launcher_args(ports: list[int], env: dict[str, str], cpu: float, memory: int) -> list[str]:
6076
"""The ``docker run`` arguments that pass the launcher KVM and its ``launcher_variables``."""
6177
return ["--device", KVM, *(arg for name, value in launcher_variables(ports, env, cpu, memory).items()
6278
for arg in ("-e", f"{name}={value}"))]
6379

6480

65-
def exported_disk(image: str, platform: str) -> Path:
66-
"""``image``'s ``DISK_FILE`` compressed with zstd in this machine's cache, exported from its image store by this
67-
machine's Docker unless an export of it is there already: once for each composed disk, by its ref, whose tag is its
68-
composer's key. ``platform`` is the disk's own, which this machine's Docker may not be."""
69-
path = cache_root() / "disks" / f"{hashlib.sha256(image.encode()).hexdigest()[:32]}.img.zst"
70-
with holding_locks([f"disk-export:{path.name}"]):
71-
if not path.exists():
72-
_export(image, platform, path)
73-
return path
81+
@dataclass(frozen=True)
82+
class Exported:
83+
"""A disk exported for a provider elsewhere to carry: its ``DISK_FILE``, compressed; for a thin disk, its base,
84+
compressed, exported once for each base; and its guest file, when it names one."""
85+
86+
disk: Path
87+
base: Path | None = None
88+
guest: Path | None = None
89+
90+
91+
def exported_disk(image: str, platform: str) -> Exported:
92+
"""``image``'s disk compressed with zstd in this machine's cache, exported from its image store by this machine's
93+
Docker unless an export of it is there already: once for each composed disk, by its ref, whose tag is its composer's
94+
key; a thin disk's base once for each base, by its layer, which every disk composed on it shares. ``platform`` is
95+
the disk's own, which this machine's Docker may not be."""
96+
cached = _Cached(cache_root() / "disks" / f"{hashlib.sha256(image.encode()).hexdigest()[:32]}.img.zst")
97+
with holding_locks([f"disk-export:{cached.disk.name}"]):
98+
if not cached.disk.exists():
99+
_export(image, platform, cached)
100+
return cached.exported()
101+
102+
103+
@dataclass(frozen=True)
104+
class _Cached:
105+
"""A disk's export in this machine's cache, named for its image's ref: the ``disk``, compressed, and beside it its
106+
guest file, a file naming its base's export, which every disk on that base shares, and the folder an export of it
107+
works in."""
108+
109+
disk: Path
74110

111+
@property
112+
def guest(self) -> Path:
113+
return self._beside("guest")
75114

76-
def _export(image: str, platform: str, path: Path) -> None:
115+
@property
116+
def base_link(self) -> Path:
117+
return self._beside("base")
118+
119+
@property
120+
def work(self) -> Path:
121+
return self._beside("work")
122+
123+
def base_of(self, layer: str) -> Path:
124+
"""The export of the base whose first layer is ``layer``."""
125+
return self.disk.with_name(f"base-{hashlib.sha256(layer.encode()).hexdigest()[:32]}.img.zst")
126+
127+
def exported(self) -> Exported:
128+
base = self.disk.with_name(self.base_link.read_text()) if self.base_link.exists() else None
129+
return Exported(self.disk, base, self.guest if self.guest.exists() else None)
130+
131+
def _beside(self, what: str) -> Path:
132+
return self.disk.with_name(f"{self.disk.name}.{what}")
133+
134+
135+
def _export(image: str, platform: str, cached: _Cached) -> None:
77136
"""Signed in to the image store first, as a boot's pull is, since ``docker create`` pulls ``image`` when it isn't
78-
here; for ``platform``, as the image's index holds that one alone, which a pull for this machine's wouldn't find."""
79-
path.parent.mkdir(parents=True, exist_ok=True)
137+
here; for ``platform``, as the image's index holds that one alone, which a pull for this machine's wouldn't find.
138+
The disk is placed last, so one that's there was exported whole, with its base and guest file. Its work folder and
139+
container are named for the export and cleared first, under its lock, so an export killed outright leaves no copy of
140+
a base behind once the disk is exported again; the container by the folder's whole path, as another user's export
141+
of the same disk through this machine's Docker holds a lock of its own."""
142+
container = f"agentenv-export-{hashlib.sha256(str(cached.work).encode()).hexdigest()[:24]}"
143+
_cleared(cached.work, container)
144+
cached.work.mkdir(parents=True)
80145
docker_login(get_config().get_image_store().auth(image))
81-
container = docker("create", "--platform", platform, image, "disk")
146+
docker("create", "--name", container, "--platform", platform, image, "disk")
82147
try:
83-
with tempfile.TemporaryDirectory(prefix="export-", dir=path.parent) as work:
84-
raw, compressed = Path(work) / "disk.img", Path(work) / path.name
85-
docker("cp", f"{container}:/{DISK_FILE}", str(raw))
86-
with raw.open("rb") as source, compressed.open("wb") as out:
87-
done = subprocess.run(["docker", "run", "--rm", "-i", "--entrypoint", "zstd", launcher(), "-q", "-3",
88-
"-T0", "-c"], stdin=source, stdout=out, stderr=subprocess.PIPE, text=False)
89-
if done.returncode:
90-
raise RuntimeError(f"compressing {image}'s disk failed: {done.stderr.decode().strip()}")
91-
os.replace(compressed, path)
148+
if (guest := _copied(container, DISK_GUEST_FILE, cached.work / "guest")) is not None:
149+
guest.replace(cached.guest)
150+
if (layer := _layout(image).base) is not None:
151+
base = cached.base_of(layer)
152+
with holding_locks([f"disk-export:{base.name}"]):
153+
if not base.exists():
154+
_compressed(container, DISK_BASE_FILE, base, cached.work)
155+
cached.base_link.write_text(base.name)
156+
_compressed(container, DISK_FILE, cached.disk, cached.work)
92157
finally:
93-
subprocess.run(["docker", "rm", container], capture_output=True)
158+
_cleared(cached.work, container)
159+
160+
161+
def _cleared(work: Path, container: str) -> None:
162+
subprocess.run(["docker", "rm", "-f", container], capture_output=True)
163+
shutil.rmtree(work, ignore_errors=True)
164+
165+
166+
def _copied(container: str, file: str, to: Path) -> Path | None:
167+
"""``file`` copied out of ``container`` to ``to``; None when the container has no such file. Any other failure is
168+
raised, as an export cached without a guest file it has would boot it as the wrong OS."""
169+
try:
170+
docker("cp", f"{container}:/{file}", str(to))
171+
except DockerFailed as e:
172+
if not any(missing in str(e) for missing in _NO_SUCH_PATH):
173+
raise
174+
return None
175+
return to
176+
177+
178+
def _compressed(container: str, file: str, path: Path, work: Path) -> None:
179+
"""``file`` of ``container`` compressed with zstd to ``path``, through ``work``."""
180+
raw, compressed = work / PurePosixPath(file).name, work / path.name
181+
docker("cp", f"{container}:/{file}", str(raw))
182+
with raw.open("rb") as source, compressed.open("wb") as out:
183+
done = subprocess.run(["docker", "run", "--rm", "-i", "--entrypoint", "zstd", launcher(), "-q", "-3", "-T0",
184+
"-c"], stdin=source, stdout=out, stderr=subprocess.PIPE, text=False)
185+
if done.returncode:
186+
raise RuntimeError(f"compressing a disk's {file} failed: {done.stderr.decode().strip()}")
187+
raw.unlink()
188+
os.replace(compressed, path)
94189

95190

96191
def launcher() -> str:

0 commit comments

Comments
 (0)