diff --git a/docs/source/getting-started.md b/docs/source/getting-started.md index e79962996..8826018a2 100644 --- a/docs/source/getting-started.md +++ b/docs/source/getting-started.md @@ -22,7 +22,7 @@ A few integrations ship as optional extras. Install them with |-------|----------| | `inspect` | The Inspect AI evaluation harness | | `harbor` | The [Harbor integration](environments/harbor) and its sandbox backends (Python 3.12+) | -| `daytona`, `aca`, `modal`, `novita` | Cloud sandbox providers (see [Runtime Providers](guides/runtime-providers)) | +| `daytona`, `aca`, `modal`, `novita`, `boxd` | Cloud sandbox providers (see [Runtime Providers](guides/runtime-providers)) | ## Try an Environment @@ -155,7 +155,7 @@ with EchoEnv(base_url="http://localhost:8000").sync() as client: result = client.reset() ``` -The same client also runs environments on cloud sandboxes (Daytona, Modal, Novita, Azure Container Apps, Hugging Face) and on a Kubernetes cluster through runtime providers. `HFSandboxProvider` ships with core OpenEnv and needs a Hugging Face token for an account that can run Jobs (the sandboxes are billed as Jobs): it runs an environment's Space image in a Hugging Face sandbox ([example](guides/runtime-providers#hfsandboxprovider)). `KubernetesProvider` is an optional extra (`pip install openenv[kubernetes]`, [example](guides/runtime-providers#kubernetesprovider)). See the [Runtime Providers guide](guides/runtime-providers) to pick one. +The same client also runs environments on cloud sandboxes (Daytona, Modal, Novita, Azure Container Apps, Hugging Face, boxd) and on a Kubernetes cluster through runtime providers. `HFSandboxProvider` ships with core OpenEnv and needs a Hugging Face token for an account that can run Jobs (the sandboxes are billed as Jobs): it runs an environment's Space image in a Hugging Face sandbox ([example](guides/runtime-providers#hfsandboxprovider)). `KubernetesProvider` is an optional extra (`pip install openenv[kubernetes]`, [example](guides/runtime-providers#kubernetesprovider)). See the [Runtime Providers guide](guides/runtime-providers) to pick one. ## Next Steps diff --git a/docs/source/guides/runtime-providers.md b/docs/source/guides/runtime-providers.md index ebf834260..3b6496e58 100644 --- a/docs/source/guides/runtime-providers.md +++ b/docs/source/guides/runtime-providers.md @@ -62,6 +62,7 @@ is a one-line change. | `NovitaSandboxProvider` | Novita AI sandboxes | `pip install openenv[novita]` | ✅ | | `HFSandboxProvider` | Hugging Face sandboxes (billed as HF Jobs) | core | ✅ | | `KubernetesProvider` | Kubernetes cluster | `pip install openenv[kubernetes]` | ✅ | +| `BoxdProvider` | boxd machines (microVMs) | `pip install openenv[boxd]` | ✅ | Cloud-provider SDKs are optional extras, imported lazily, so installing core OpenEnv pulls in no cloud SDK. The core providers (`LocalDockerProvider`, @@ -91,8 +92,8 @@ async with MyEnv(provider=provider) as env: ... ``` -`ModalProvider`, `DaytonaProvider`, `ACASandboxProvider`, and `HFSandboxProvider` -support this provider-owned flow. Providers that require an explicit image at +`ModalProvider`, `DaytonaProvider`, `ACASandboxProvider`, `HFSandboxProvider`, +and `BoxdProvider` support this provider-owned flow. Providers that require an explicit image at `start_container()` time, such as `LocalDockerProvider` and `DockerSwarmProvider`, should still be started manually and passed in with the returned `base_url`: @@ -178,6 +179,50 @@ provider = ACASandboxProvider( ) ``` +### BoxdProvider + +Runs the server in a [boxd](https://boxd.sh?ref=OpenEnv) machine, a microVM +with Docker installed. Install with `pip install openenv[boxd]`. Requires a +boxd API key, as `api_key=` or the `BOXD_API_KEY` environment variable +(optionally `org=`/`BOXD_ORG` and `region=`/`BOXD_REGION`, `fra` or `nyc`). + +The provider creates one isolated machine per `start_container`, runs the +image in it with `docker run`, and returns the machine's HTTPS URL. The URL is +reachable from the internet with no login in front of it, so treat it like a +secret: anyone who has it can drive the environment. The provider keeps it out +of its errors. +`stop_container()` deletes the machine. `env_vars` reach the container through +a `docker run --env-file` that is removed after the run. + +```python +from openenv.core.containers.runtime.boxd_provider import BoxdProvider + +provider = BoxdProvider(image="ghcr.io/huggingface/openenv-echo-env:latest") +``` + +A local Dockerfile works too. The build context is uploaded to the machine and +built there with `docker build`: + +```python +image = BoxdProvider.image_from_dockerfile("envs/echo_env/server/Dockerfile") +provider = BoxdProvider(image=image) +``` + +boxd also deletes the machine `delete_after` seconds after it was created, one +hour by default, even when `stop_container()` never runs (a crashed script, or +a Ctrl-C while the machine is being created). Set it above your longest +session, or pass `delete_after=None` to keep the machine until +`stop_container()`: + +```python +provider = BoxdProvider(image="ghcr.io/huggingface/openenv-echo-env:latest", delete_after=4 * 3600) +``` + +`vcpu` and `memory` pick the machine size (`1`/`"4G"`, `2`/`"8G"`, `4`/`"16G"` +or `8`/`"32G"`); the organization's default applies otherwise. + +Full example: [`examples/boxd_echo_env.py`](https://github.com/openenv/OpenEnv/blob/main/examples/boxd_echo_env.py). + ### DaytonaProvider Runs the server in a Daytona cloud sandbox. Install with diff --git a/docs/source/reference/core.md b/docs/source/reference/core.md index fc0b67269..f7d9138a6 100644 --- a/docs/source/reference/core.md +++ b/docs/source/reference/core.md @@ -293,3 +293,5 @@ See [Evaluating with Environments](../tutorials/evaluation-inspect) for a worked [[autodoc]] openenv.core.containers.runtime.hf_sandbox_provider.HFSandboxProvider [[autodoc]] openenv.core.containers.runtime.kubernetes_provider.KubernetesProvider + +[[autodoc]] openenv.core.containers.runtime.boxd_provider.BoxdProvider diff --git a/examples/boxd_echo_env.py b/examples/boxd_echo_env.py new file mode 100644 index 000000000..5f12d60c7 --- /dev/null +++ b/examples/boxd_echo_env.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +"""Hello-world example running the Echo environment on boxd. + +Creates a boxd machine via ``BoxdProvider``, runs the echo-env image in it with +Docker, then talks to it through ``EchoEnv`` over the machine's HTTPS URL +(https/wss). The machine is deleted when the provider closes. + +Usage: + pip install "openenv[boxd]" + export BOXD_API_KEY=... (https://boxd.sh?ref=OpenEnv) + PYTHONPATH=src:envs uv run python examples/boxd_echo_env.py + +Optional environment variables: + OPENENV_IMAGE registry image, or "dockerfile" to build + envs/echo_env/server/Dockerfile in the machine + (default: ghcr.io/huggingface/openenv-echo-env:latest) + BOXD_REGION "fra" or "nyc" (read by the boxd SDK) + OPENENV_NAME_PREFIX machine name prefix (default: openenv) +""" + +import asyncio +import logging +import os + +from echo_env import EchoEnv +from openenv.core.containers.runtime.boxd_provider import BoxdProvider + +logger = logging.getLogger(__name__) + + +async def _interact(base_url: str) -> None: + """Run the async WebSocket interaction against the running machine.""" + async with EchoEnv(base_url=base_url) as env: + await env.reset() + + tools = await env.list_tools() + logger.info("Available tools: %s", [t.name for t in tools]) + + echoed = await env.call_tool("echo_message", message="Hello, World!") + logger.info("echo_message -> %s", echoed) + + +def main() -> int: + logging.basicConfig(level=logging.INFO, format="%(message)s") + + image = os.environ.get( + "OPENENV_IMAGE", "ghcr.io/huggingface/openenv-echo-env:latest" + ) + if image == "dockerfile": + image = BoxdProvider.image_from_dockerfile("envs/echo_env/server/Dockerfile") + + with BoxdProvider( + name_prefix=os.environ.get("OPENENV_NAME_PREFIX", "openenv"), + ) as provider: + logger.info("Starting boxd machine...") + base_url = provider.start_container(image) + logger.info("Machine %s is up", provider.machine_id) + provider.wait_for_ready(base_url, timeout_s=180) + logger.info("Server ready.") + asyncio.run(_interact(base_url)) + logger.info("Deleting machine...") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/pyproject.toml b/pyproject.toml index 780dd0ae1..eebe94298 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -60,6 +60,9 @@ kubernetes = [ novita = [ "novita-sandbox>=2.1.1", ] +boxd = [ + "boxd>=0.3.1", +] inspect = [ "inspect-ai>=0.3.0", ] diff --git a/src/openenv/core/containers/runtime/boxd_provider.py b/src/openenv/core/containers/runtime/boxd_provider.py new file mode 100644 index 000000000..ac0fc3912 --- /dev/null +++ b/src/openenv/core/containers/runtime/boxd_provider.py @@ -0,0 +1,487 @@ +# SPDX-License-Identifier: BSD-3-Clause + +""" +boxd container provider for running OpenEnv environments in boxd machines. + +Requires the ``boxd`` SDK: ``pip install boxd>=0.3.1`` + +The provider creates a boxd machine (a microVM with Docker installed), runs the +environment image in it with ``docker run``, and returns the machine's +``https://`` URL that ``EnvClient`` connects to over ``wss://``. +""" + +from __future__ import annotations + +import io +import logging +import os +import pathlib +import re +import shlex +import tarfile +import time +import uuid +from typing import Any, Dict + +from .providers import ContainerProvider + +logger = logging.getLogger(__name__) + +_PORT = 8000 +_CONTAINER = "openenv-server" +_BUILD_DIR = "/home/boxd/openenv-build" +_BUILT_IMAGE = "openenv-env:latest" +_ENV_DIR = "/home/boxd/openenv-env" +_ENV_FILE = f"{_ENV_DIR}/server.env" +_ENV_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +_SKIP_IN_CONTEXT = {".git", ".venv", "__pycache__"} + + +def _require_secure_url(url: Any) -> str: + """Refuse a base URL that is not https, so ``EnvClient`` never falls back + to plaintext ``ws://``. The URL itself stays out of the error.""" + if not isinstance(url, str) or not url.lower().startswith("https://"): + raise RuntimeError( + "The boxd machine returned a non-HTTPS URL. OpenEnv requires an " + "https/wss base_url, so the provider refuses to connect." + ) + return url + + +class BoxdProvider(ContainerProvider): + """ + Container provider that runs environments in [boxd](https://boxd.sh?ref=OpenEnv) + machines. + + Each ``start_container`` creates one isolated machine, runs the environment + image in it with Docker, and returns the machine's public ``https://`` URL. + ``image`` is either a registry reference (``"ghcr.io/org/echo-env:latest"``) + or a ``"dockerfile:"`` reference returned by + [`~BoxdProvider.image_from_dockerfile`], which is built inside the machine. + + The environment runs untrusted code, so the machine is always created + isolated: it joins no private network and holds no boxd credentials. + Server output is kept out of raised errors unless + ``surface_server_logs=True``. + + Only one machine is active per provider: calling ``start_container`` again + before ``stop_container()``/``close()`` raises ``RuntimeError``. + ``stop_container()`` deletes the machine. + + As a backstop, boxd deletes the machine ``delete_after`` seconds after it + was created (one hour by default), even if ``stop_container()`` never + runs. This also covers a Ctrl-C during the create call itself, before the + provider knows the machine id and can delete it. + + Examples: + + ```python + with BoxdProvider(region="fra") as provider: + base_url = provider.start_container("ghcr.io/huggingface/openenv-echo-env:latest") + provider.wait_for_ready(base_url) + # machine deleted on exit + ``` + """ + + _dockerfile_registry: Dict[str, Dict[str, str]] = {} + + def __init__( + self, + *, + image: str | None = None, + env_vars: dict[str, str] | None = None, + api_key: str | None = None, + org: str | None = None, + region: str | None = None, + vcpu: int | None = None, + memory: str | None = None, + name_prefix: str = "openenv", + delete_after: int | None = 3600, + start_timeout: float = 900.0, + surface_server_logs: bool = False, + _client: Any = None, + ): + """ + Args: + image (`str`, *optional*): + Registry image or ``"dockerfile:"`` source to use when + ``start_container()`` is called without an image. + env_vars (`dict`, *optional*): + Environment variables to use when ``start_container()`` is + called without explicit ``env_vars``. + api_key (`str`, *optional*): + boxd API key. Falls back to the ``BOXD_API_KEY`` env var. + org (`str`, *optional*): + boxd organization to create the machine in. Falls back to + ``BOXD_ORG``, then to the key's own organization. + region (`str`, *optional*): + boxd region, ``"fra"`` or ``"nyc"``. Falls back to + ``BOXD_REGION``, then to the account default. + vcpu (`int`, *optional*): + vCPUs for the machine (``1``, ``2``, ``4`` or ``8``). When + `None`, the organization's default size applies. + memory (`str`, *optional*): + Memory for the machine, matching ``vcpu`` (``"4G"``, ``"8G"``, + ``"16G"`` or ``"32G"``). + name_prefix (`str`, *optional*, defaults to `"openenv"`): + Prefix for the machine name. A random suffix is appended. + delete_after (`int`, *optional*, defaults to `3600`): + Seconds after creation when boxd deletes the machine, whatever + it is doing. Set it above the longest session you run. `None` + turns the timer off, so the machine lives until + ``stop_container()``. + start_timeout (`float`, *optional*, defaults to `900.0`): + Seconds allowed for each of ``docker build`` (for a Dockerfile + image) and ``docker run`` (which pulls a registry image). + surface_server_logs (`bool`, *optional*, defaults to `False`): + When `True`, a failed ``docker build`` includes the tail of + its output, and startup-crash errors include the tail of the + container's output with injected env-var values redacted. + """ + if delete_after is not None and ( + isinstance(delete_after, bool) + or not isinstance(delete_after, int) + or delete_after <= 0 + ): + raise ValueError( + f"delete_after must be a positive number of seconds or None " + f"(got {delete_after!r})." + ) + if _client is None: + from boxd import Boxd + + _client = Boxd(api_key=api_key, region=region) + self._client = _client + self._image = image + self._env_vars = env_vars + self._org = org or os.environ.get("BOXD_ORG") or None + self._vcpu = vcpu + self._memory = memory + self._name_prefix = name_prefix + self._delete_after = delete_after + self._start_timeout = start_timeout + self._surface_server_logs = surface_server_logs + self._machine_id: str | None = None + self._base_url: str | None = None + self._redact_values: set[str] = set() + + @classmethod + def image_from_dockerfile( + cls, + dockerfile_path: str, + context_dir: str | None = None, + ) -> str: + """Validate a Dockerfile and return a ``dockerfile:`` URI for + [`~BoxdProvider.start_container`]. + + The build context is uploaded to the machine and built there with + ``docker build`` when the provider starts. + + Args: + dockerfile_path (`str`): + Path to the Dockerfile on disk. + context_dir (`str`, *optional*): + Build context directory. Defaults to the Dockerfile's + grandparent directory, matching the ``openenv init`` layout + where Dockerfiles live in ``/server/Dockerfile`` and the + build context is ``/``. + + Returns: + `str`: A ``"dockerfile:"`` string to pass to + ``start_container``. + + Raises: + FileNotFoundError: If *dockerfile_path* does not exist. + ValueError: If *context_dir* is given but does not exist, or if + COPY sources in the Dockerfile cannot be found under it. + """ + src = pathlib.Path(dockerfile_path).resolve() + if not src.is_file(): + raise FileNotFoundError(f"Dockerfile not found: {dockerfile_path}") + + if context_dir is not None: + ctx = pathlib.Path(context_dir).resolve() + if not ctx.is_dir(): + raise ValueError(f"context_dir does not exist: {context_dir}") + else: + ctx = src.parent.parent + + for line in src.read_text().splitlines(): + m = re.match(r"^\s*COPY\s+(?!--from=)(\S+)\s+", line, re.IGNORECASE) + if not m or m.group(1).startswith("/"): + continue + copy_src = m.group(1) + if not (ctx / copy_src).exists() and not any(ctx.glob(copy_src)): + raise ValueError( + f"Dockerfile COPY source '{copy_src}' not found under " + f"context_dir '{ctx}'. Pass context_dir explicitly." + ) + + cls._dockerfile_registry[str(src)] = { + "dockerfile_path": str(src), + "context_dir": str(ctx), + } + return f"dockerfile:{src}" + + def _exec(self, command: str, *, timeout: float = 30.0, env=None) -> Any: + return self._client.machines.exec( + self._machine_id, command, env=env, timeout=timeout + ) + + def _build_image(self, meta: Dict[str, str]) -> str: + """Upload the registered build context and build it in the machine.""" + buf = io.BytesIO() + with tarfile.open(fileobj=buf, mode="w:gz") as tar: + tar.add( + meta["context_dir"], + arcname=".", + filter=lambda info: ( + None + if _SKIP_IN_CONTEXT & set(pathlib.PurePath(info.name).parts) + else info + ), + ) + files = self._client.machines.files + files.upload(self._machine_id, f"{_BUILD_DIR}.tar.gz", buf.getvalue()) + files.upload( + self._machine_id, + f"{_BUILD_DIR}.Dockerfile", + pathlib.Path(meta["dockerfile_path"]).read_bytes(), + ) + + result = self._exec( + f"mkdir -p {_BUILD_DIR} && tar -xzf {_BUILD_DIR}.tar.gz -C {_BUILD_DIR}" + f" && docker build --progress=plain -t {_BUILT_IMAGE}" + f" -f {_BUILD_DIR}.Dockerfile {_BUILD_DIR}", + timeout=self._start_timeout, + ) + if result.exit_code != 0: + message = ( + f"docker build failed in the boxd machine (exit {result.exit_code})." + ) + if self._surface_server_logs: + message += f"\n{(result.stdout + result.stderr)[-2000:]}" + else: + message += ( + " Its output is not surfaced; construct the provider with " + "surface_server_logs=True to include it." + ) + raise RuntimeError(message) + return _BUILT_IMAGE + + def start_container( + self, + image: str | None = None, + port: int | None = None, + env_vars: dict[str, str] | None = None, + **kwargs: Any, + ) -> str: + """ + Create a boxd machine and run the environment image in it. + + Args: + image (`str`, *optional*): + Registry image (e.g. ``"ghcr.io/org/echo-env:latest"``) or + ``"dockerfile:"`` returned by + [`~BoxdProvider.image_from_dockerfile`]. May be omitted when + supplied to the constructor. + port (`int`, *optional*): + Must be `None` or `8000`. The machine's HTTPS URL routes to + port 8000; other ports raise ``ValueError``. + env_vars (`dict`, *optional*): + Environment variables passed to the container. Names must be + shell identifiers and values must be single lines. + **kwargs: + Not supported; any keyword raises ``TypeError``. + + Returns: + `str`: The machine's HTTPS URL (base_url). + """ + if self._machine_id is not None: + raise RuntimeError( + "BoxdProvider already has an active machine. Call " + "stop_container() (or close()) before starting another." + ) + if port is not None and port != _PORT: + raise ValueError( + f"BoxdProvider only supports port {_PORT} (got {port}). The " + f"machine's HTTPS URL routes to port {_PORT}." + ) + if kwargs: + raise TypeError(f"Unsupported start_container options: {sorted(kwargs)}") + + effective_image = image if image is not None else self._image + if effective_image is None: + raise ValueError( + "BoxdProvider requires an image. Pass it to the constructor " + "or start_container()." + ) + build_meta = None + if effective_image.startswith("dockerfile:"): + dockerfile_path = effective_image[len("dockerfile:") :] + build_meta = self._dockerfile_registry.get(dockerfile_path) + if build_meta is None: + raise ValueError( + f"No registered Dockerfile metadata for {dockerfile_path}. " + "Call BoxdProvider.image_from_dockerfile() first." + ) + effective_env_vars = dict( + (self._env_vars if env_vars is None else env_vars) or {} + ) + for name, value in effective_env_vars.items(): + if not _ENV_NAME.match(name) or "\n" in value: + raise ValueError( + f"Invalid env var {name!r}: names must be shell identifiers " + "and values must be single lines." + ) + self._redact_values = {v for v in effective_env_vars.values() if v} + + machine = self._client.machines.create( + f"{self._name_prefix}-{uuid.uuid4().hex[:12]}", + org=self._org, + isolated=True, + labels={"managed-by": "openenv"}, + vcpu=self._vcpu, + memory=self._memory, + delete_after=self._delete_after, + ) + self._machine_id = machine.id + + try: + ready = self._client.machines.wait_until_ready(machine.id) + url = _require_secure_url(ready.access.url) + # EnvClient is not a browser, so the browser check on the + # machine's URL is turned off. + self._client.machines.set_bot_protection(machine.id, mode="off") + + if build_meta is not None: + effective_image = self._build_image(build_meta) + + # Env values go in a file read by `docker run --env-file`, so they + # never appear on a command line. The file is written into a + # directory only its owner can read, and removed after the run. + env_flag = "" + if effective_env_vars: + mkdir = self._exec(f"install -d -m 700 {_ENV_DIR}") + if mkdir.exit_code != 0: + raise RuntimeError( + f"Could not create {_ENV_DIR} in the boxd machine " + f"(exit {mkdir.exit_code}): {mkdir.stderr[-500:]}" + ) + self._client.machines.files.upload( + machine.id, + _ENV_FILE, + "".join(f"{k}={v}\n" for k, v in effective_env_vars.items()), + ) + env_flag = f" --env-file {_ENV_FILE}" + run = ( + f"chmod 600 {_ENV_FILE} 2>/dev/null; docker run -d --name {_CONTAINER}" + f"{env_flag} -p {_PORT}:{_PORT} {shlex.quote(effective_image)};" + f" rc=$?; rm -rf {_ENV_DIR}; exit $rc" + ) + result = self._exec(run, timeout=self._start_timeout) + if result.exit_code != 0: + raise RuntimeError( + f"docker run failed in the boxd machine (exit {result.exit_code}):" + f"\n{result.stderr[-2000:]}" + ) + self._base_url = url + except BaseException: + try: + self.stop_container() + except Exception: + logger.warning( + "Could not delete boxd machine %s after a failed start; " + "delete it with stop_container() or in the boxd console.", + machine.id, + ) + raise + + return self._base_url + + def stop_container(self) -> None: + """Delete the boxd machine. + + The machine id is kept when the delete fails, so calling this again + retries it. A machine that is already gone counts as deleted. + """ + if self._machine_id is None: + self._redact_values = set() + return + try: + self._client.machines.delete(self._machine_id) + except Exception as exc: + from boxd.errors import NotFoundError + + if not isinstance(exc, NotFoundError): + raise + self._machine_id = None + self._base_url = None + self._redact_values = set() + + def close(self) -> None: + """Delete the active machine and close the SDK client.""" + try: + self.stop_container() + finally: + self._client.close() + + @property + def machine_id(self) -> str: + """Id of the boxd machine started by the last ``start_container``.""" + if self._machine_id is None: + raise RuntimeError("BoxdProvider has no active machine.") + return self._machine_id + + def _server_died_message(self) -> str: + base = "The environment container in the boxd machine exited during startup." + if not self._surface_server_logs: + return ( + f"{base} Its output is not surfaced to avoid leaking injected " + "secrets; construct the provider with surface_server_logs=True " + "to include a redacted excerpt." + ) + log = self._exec(f"docker logs --tail 100 {_CONTAINER} 2>&1").stdout + for value in self._redact_values: + log = log.replace(value, "***") + return f"{base}\nLog (redacted):\n{log[-2000:]}" + + def wait_for_ready(self, base_url: str, timeout_s: float = 120.0) -> None: + """ + Poll the /health endpoint until the environment server answers. + + Args: + base_url (`str`): + URL returned by ``start_container()``. + timeout_s (`float`, *optional*, defaults to `120.0`): + Maximum seconds to wait. + + Raises: + TimeoutError: If the server doesn't become ready in time. + RuntimeError: If the container exited. + """ + import requests + + health_url = f"{base_url}/health" + deadline = time.time() + timeout_s + while time.time() < deadline: + try: + response = requests.get(health_url, timeout=5.0) + if response.status_code == 200: + return + except requests.RequestException: + pass + + if self._machine_id is not None: + state = self._exec( + f"docker inspect -f '{{{{.State.Status}}}}' {_CONTAINER}" + ).stdout.strip() + if state in ("exited", "dead"): + raise RuntimeError(self._server_died_message()) + + time.sleep(1.0) + + raise TimeoutError(f"The boxd machine did not become ready within {timeout_s}s") + + +__all__ = ["BoxdProvider"] diff --git a/tests/test_core/test_boxd_provider.py b/tests/test_core/test_boxd_provider.py new file mode 100644 index 000000000..bf6e39a1b --- /dev/null +++ b/tests/test_core/test_boxd_provider.py @@ -0,0 +1,499 @@ +# SPDX-License-Identifier: BSD-3-Clause + +"""Unit tests for BoxdProvider. + +The tests inject a duck-typed fake client (``_client=``), so they run without +``pip install boxd``. One test installs a minimal fake ``boxd`` module to pin +how the real client is constructed. +""" + +from __future__ import annotations + +import io +import sys +import tarfile +import types +from types import SimpleNamespace +from unittest.mock import MagicMock, patch + +import pytest +from openenv.core.containers.runtime.boxd_provider import BoxdProvider +from openenv.core.containers.runtime.providers import ContainerProvider + +ECHO = "ghcr.io/huggingface/openenv-echo-env:latest" +URL = "https://openenv-abc.fra.boxd.sh" + + +class _FakeMachines: + def __init__(self): + self.created: list[dict] = [] + self.deleted: list[str] = [] + self.bot_protection: list[tuple[str, str]] = [] + self.commands: list[str] = [] + self.exec_envs: list[dict | None] = [] + self.uploads: dict[str, bytes] = {} + self.commands_before_upload: dict[str, list[str]] = {} + self.files = SimpleNamespace(upload=self._upload) + self.fail_ready = False + self.url = URL + self.delete_error: Exception | None = None + self.run_exit = 0 + self.build_exit = 0 + self.container_state = "running" + self.logs = "" + + def create(self, name, **kwargs): + self.created.append({"name": name, **kwargs}) + return SimpleNamespace(id="m-1", name=name) + + def wait_until_ready(self, machine_id): + if self.fail_ready: + raise TimeoutError("not ready") + return SimpleNamespace(id=machine_id, access=SimpleNamespace(url=self.url)) + + def set_bot_protection(self, machine_id, *, mode): + self.bot_protection.append((machine_id, mode)) + + def _upload(self, machine_id, path, data): + self.uploads[path] = data + self.commands_before_upload[path] = list(self.commands) + return len(data) + + def exec(self, machine_id, command, *, env=None, timeout=None): + self.commands.append(command) + self.exec_envs.append(env) + if "docker run" in command: + return SimpleNamespace(exit_code=self.run_exit, stdout="", stderr="denied") + if "docker build" in command: + return SimpleNamespace( + exit_code=self.build_exit, stdout="step 1/3", stderr="build broke" + ) + if command.startswith("docker inspect"): + return SimpleNamespace( + exit_code=0, stdout=self.container_state + "\n", stderr="" + ) + if command.startswith("docker logs"): + return SimpleNamespace(exit_code=0, stdout=self.logs, stderr="") + return SimpleNamespace(exit_code=0, stdout="", stderr="") + + def delete(self, machine_id): + if self.delete_error is not None: + raise self.delete_error + self.deleted.append(machine_id) + + +class _FakeClient: + def __init__(self): + self.machines = _FakeMachines() + self.closed = 0 + + def close(self): + self.closed += 1 + + +@pytest.fixture +def client(): + return _FakeClient() + + +def _provider(client, **kwargs): + return BoxdProvider(_client=client, **kwargs) + + +def test_is_a_container_provider(client): + assert isinstance(_provider(client), ContainerProvider) + + +def test_start_creates_an_isolated_machine_and_returns_its_url(client): + provider = _provider(client, org="acme", vcpu=4, memory="16G") + + assert provider.start_container(ECHO) == URL + + created = client.machines.created[0] + assert created["isolated"] is True + assert created["org"] == "acme" + assert created["vcpu"] == 4 and created["memory"] == "16G" + assert created["labels"] == {"managed-by": "openenv"} + assert created["name"].startswith("openenv-") + assert provider.machine_id == "m-1" + + +def test_machine_gets_a_one_hour_delete_after_by_default(client): + _provider(client).start_container(ECHO) + assert client.machines.created[0]["delete_after"] == 3600 + + +def test_delete_after_is_configurable_and_can_be_turned_off(client): + _provider(client, delete_after=600).start_container(ECHO) + other = _FakeClient() + _provider(other, delete_after=None).start_container(ECHO) + assert client.machines.created[0]["delete_after"] == 600 + assert other.machines.created[0]["delete_after"] is None + + +@pytest.mark.parametrize("delete_after", [0, -5, 1.5, True, "1h"]) +def test_invalid_delete_after_is_rejected(client, delete_after): + with pytest.raises(ValueError, match="delete_after"): + _provider(client, delete_after=delete_after) + + +def test_failed_env_dir_deletes_the_machine(client): + original = client.machines.exec + + def exec_(machine_id, command, **kwargs): + if command.startswith("install -d"): + client.machines.commands.append(command) + return SimpleNamespace(exit_code=1, stdout="", stderr="no space") + return original(machine_id, command, **kwargs) + + client.machines.exec = exec_ + with pytest.raises(RuntimeError, match="no space"): + _provider(client).start_container(ECHO, env_vars={"A": "1"}) + assert client.machines.deleted == ["m-1"] + assert client.machines.uploads == {} + + +def test_name_prefix(client): + _provider(client, name_prefix="rollout").start_container(ECHO) + assert client.machines.created[0]["name"].startswith("rollout-") + + +def test_org_falls_back_to_env(client): + with patch.dict("os.environ", {"BOXD_ORG": "from-env"}): + _provider(client).start_container(ECHO) + assert client.machines.created[0]["org"] == "from-env" + + +def test_bot_protection_is_turned_off(client): + _provider(client).start_container(ECHO) + assert client.machines.bot_protection == [("m-1", "off")] + + +def test_docker_run_publishes_port_8000(client): + _provider(client).start_container(ECHO) + run = _run_command(client) + assert "-p 8000:8000" in run + assert "--name openenv-server" in run + assert f" {ECHO};" in run + + +def _run_command(client): + return next(c for c in client.machines.commands if "docker run" in c) + + +def test_env_values_go_in_an_env_file_not_on_a_command_line(client): + _provider(client).start_container(ECHO, env_vars={"API_TOKEN": "s3cret"}) + + env_file = "/home/boxd/openenv-env/server.env" + assert client.machines.uploads[env_file] == "API_TOKEN=s3cret\n" + # The file lands in a directory only its owner can read, created first. + assert client.machines.commands_before_upload[env_file] == [ + "install -d -m 700 /home/boxd/openenv-env" + ] + run = _run_command(client) + assert f"--env-file {env_file}" in run + assert "rm -rf /home/boxd/openenv-env" in run + assert "s3cret" not in "".join(client.machines.commands) + # The SDK turns exec env into a prefix of the command text, so the + # provider never passes env to exec. + assert client.machines.exec_envs == [None] * len(client.machines.exec_envs) + + +def test_no_env_file_without_env_vars(client): + _provider(client).start_container(ECHO) + assert "--env-file" not in _run_command(client) + assert client.machines.uploads == {} + + +@pytest.mark.parametrize( + "env_vars", + [{"A;touch /tmp/x": "1"}, {"MY-VAR": "1"}, {"OK": "two\nlines"}], +) +def test_invalid_env_vars_are_rejected_before_create(client, env_vars): + with pytest.raises(ValueError, match="Invalid env var"): + _provider(client).start_container(ECHO, env_vars=env_vars) + assert client.machines.created == [] + + +def test_constructor_image_and_env_vars_are_defaults(client): + provider = _provider(client, image=ECHO, env_vars={"A": "1"}) + provider.start_container() + assert _run_command(client).endswith( + f"{ECHO}; rc=$?; rm -rf /home/boxd/openenv-env; exit $rc" + ) + assert client.machines.uploads["/home/boxd/openenv-env/server.env"] == "A=1\n" + + +def test_requires_an_image(client): + with pytest.raises(ValueError, match="requires an image"): + _provider(client).start_container() + assert client.machines.created == [] + + +@pytest.mark.parametrize("port", [None, 8000]) +def test_port_8000_is_accepted(client, port): + assert _provider(client).start_container(ECHO, port=port) == URL + + +def test_other_ports_are_rejected(client): + with pytest.raises(ValueError, match="only supports port 8000"): + _provider(client).start_container(ECHO, port=9000) + assert client.machines.created == [] + + +def test_unknown_options_are_rejected(client): + with pytest.raises(TypeError, match="Unsupported"): + _provider(client).start_container(ECHO, gpu=True) + assert client.machines.created == [] + + +def test_second_start_raises_instead_of_orphaning(client): + provider = _provider(client) + provider.start_container(ECHO) + with pytest.raises(RuntimeError, match="already has an active machine"): + provider.start_container(ECHO) + assert len(client.machines.created) == 1 + + +def test_failed_docker_run_deletes_the_machine(client): + client.machines.run_exit = 125 + provider = _provider(client) + with pytest.raises(RuntimeError, match="docker run failed"): + provider.start_container(ECHO) + assert client.machines.deleted == ["m-1"] + with pytest.raises(RuntimeError, match="no active machine"): + provider.machine_id + + +def test_failed_readiness_deletes_the_machine(client): + client.machines.fail_ready = True + with pytest.raises(TimeoutError): + _provider(client).start_container(ECHO) + assert client.machines.deleted == ["m-1"] + + +def test_stop_deletes_by_id_and_is_idempotent(client): + provider = _provider(client) + provider.start_container(ECHO) + provider.stop_container() + provider.stop_container() + assert client.machines.deleted == ["m-1"] + + +def test_close_deletes_and_closes_the_client(client): + with _provider(client) as provider: + provider.start_container(ECHO) + assert client.machines.deleted == ["m-1"] + assert client.closed == 1 + + +class _NotFoundError(Exception): + pass + + +@pytest.fixture +def fake_boxd_errors(): + errors = types.ModuleType("boxd.errors") + errors.NotFoundError = _NotFoundError + boxd = types.ModuleType("boxd") + boxd.errors = errors + with patch.dict(sys.modules, {"boxd": boxd, "boxd.errors": errors}): + yield + + +def test_failed_delete_keeps_the_id_so_stop_can_retry(client, fake_boxd_errors): + provider = _provider(client) + provider.start_container(ECHO) + client.machines.delete_error = ConnectionError("network down") + with pytest.raises(ConnectionError): + provider.stop_container() + assert provider.machine_id == "m-1" + + client.machines.delete_error = None + provider.stop_container() + assert client.machines.deleted == ["m-1"] + with pytest.raises(RuntimeError, match="no active machine"): + provider.machine_id + + +def test_already_deleted_machine_counts_as_stopped(client, fake_boxd_errors): + provider = _provider(client) + provider.start_container(ECHO) + client.machines.delete_error = _NotFoundError("gone") + provider.stop_container() + with pytest.raises(RuntimeError, match="no active machine"): + provider.machine_id + + +def test_failed_start_with_failed_cleanup_raises_the_start_error( + client, fake_boxd_errors, caplog +): + client.machines.run_exit = 125 + client.machines.delete_error = ConnectionError("network down") + provider = _provider(client) + with pytest.raises(RuntimeError, match="docker run failed"): + provider.start_container(ECHO) + assert "m-1" in caplog.text + assert provider.machine_id == "m-1" + + +def test_interrupted_start_deletes_the_machine(client): + client.machines.set_bot_protection = MagicMock(side_effect=KeyboardInterrupt) + with pytest.raises(KeyboardInterrupt): + _provider(client).start_container(ECHO) + assert client.machines.deleted == ["m-1"] + + +def _health(status): + response = MagicMock() + response.status_code = status + return response + + +def test_wait_for_ready_returns_on_healthy(client): + provider = _provider(client) + url = provider.start_container(ECHO) + with patch("requests.get", return_value=_health(200)) as get: + provider.wait_for_ready(url, timeout_s=5) + get.assert_called_once_with(f"{URL}/health", timeout=5.0) + + +def test_wait_for_ready_raises_when_the_container_exited(client): + client.machines.container_state = "exited" + client.machines.logs = "token=s3cret crashed" + provider = _provider(client) + url = provider.start_container(ECHO, env_vars={"T": "s3cret"}) + with patch("requests.get", return_value=_health(502)): + with pytest.raises(RuntimeError, match="exited during startup") as err: + provider.wait_for_ready(url, timeout_s=5) + assert "s3cret" not in str(err.value) + assert "crashed" not in str(err.value) + + +def test_surface_server_logs_includes_redacted_output(client): + client.machines.container_state = "exited" + client.machines.logs = "token=s3cret crashed" + provider = _provider(client, surface_server_logs=True) + url = provider.start_container(ECHO, env_vars={"T": "s3cret"}) + with patch("requests.get", return_value=_health(502)): + with pytest.raises(RuntimeError) as err: + provider.wait_for_ready(url, timeout_s=5) + assert "token=*** crashed" in str(err.value) + + +def test_wait_for_ready_times_out(client): + provider = _provider(client) + url = provider.start_container(ECHO) + with ( + patch("requests.get", return_value=_health(502)), + patch("time.sleep"), + patch("time.time", side_effect=[0.0, 0.0, 10.0]), + ): + with pytest.raises(TimeoutError, match="did not become ready") as err: + provider.wait_for_ready(url, timeout_s=5) + # The URL is the environment's address, so it stays out of errors. + assert url not in str(err.value) + + +def test_non_https_url_is_refused_and_the_machine_deleted(client): + client.machines.url = "http://openenv-abc.fra.boxd.sh" + with pytest.raises(RuntimeError, match="non-HTTPS") as err: + _provider(client).start_container(ECHO) + assert "openenv-abc" not in str(err.value) + assert client.machines.deleted == ["m-1"] + assert not any("docker run" in c for c in client.machines.commands) + + +def _env_dir(tmp_path): + env = tmp_path / "my_env" + (env / "server").mkdir(parents=True) + (env / "server" / "Dockerfile").write_text( + 'FROM python:3.11\nCOPY . /app/env\nCMD ["server"]\n' + ) + (env / "app.py").write_text("print('hi')\n") + (env / ".venv").mkdir() + (env / ".venv" / "big").write_text("x") + return env + + +def test_image_from_dockerfile_returns_a_uri(tmp_path): + env = _env_dir(tmp_path) + uri = BoxdProvider.image_from_dockerfile(str(env / "server" / "Dockerfile")) + assert uri == f"dockerfile:{(env / 'server' / 'Dockerfile').resolve()}" + + +def test_image_from_dockerfile_missing_file(tmp_path): + with pytest.raises(FileNotFoundError): + BoxdProvider.image_from_dockerfile(str(tmp_path / "Dockerfile")) + + +def test_image_from_dockerfile_bad_context(tmp_path): + env = _env_dir(tmp_path) + with pytest.raises(ValueError, match="context_dir does not exist"): + BoxdProvider.image_from_dockerfile( + str(env / "server" / "Dockerfile"), context_dir=str(tmp_path / "nope") + ) + + +def test_image_from_dockerfile_missing_copy_source(tmp_path): + env = _env_dir(tmp_path) + (env / "server" / "Dockerfile").write_text("FROM python:3.11\nCOPY missing /x\n") + with pytest.raises(ValueError, match="COPY source 'missing'"): + BoxdProvider.image_from_dockerfile(str(env / "server" / "Dockerfile")) + + +def test_dockerfile_image_is_built_in_the_machine(client, tmp_path): + env = _env_dir(tmp_path) + image = BoxdProvider.image_from_dockerfile(str(env / "server" / "Dockerfile")) + + _provider(client).start_container(image) + + uploads = client.machines.uploads + dockerfile = uploads["/home/boxd/openenv-build.Dockerfile"] + assert dockerfile.startswith(b"FROM python:3.11") + tar = tarfile.open( + fileobj=io.BytesIO(uploads["/home/boxd/openenv-build.tar.gz"]), mode="r:gz" + ) + names = tar.getnames() + assert "./app.py" in names and "./server/Dockerfile" in names + assert not any(".venv" in n for n in names) + + commands = client.machines.commands + build = next(i for i, c in enumerate(commands) if "docker build" in c) + run = next(i for i, c in enumerate(commands) if "docker run" in c) + assert build < run + assert " openenv-env:latest;" in commands[run] + + +def test_failed_build_deletes_the_machine(client, tmp_path): + env = _env_dir(tmp_path) + image = BoxdProvider.image_from_dockerfile(str(env / "server" / "Dockerfile")) + client.machines.build_exit = 1 + with pytest.raises(RuntimeError, match="docker build failed") as err: + _provider(client).start_container(image) + assert client.machines.deleted == ["m-1"] + # Build output comes from the environment's own RUN steps: opt-in only. + assert "build broke" not in str(err.value) + assert "surface_server_logs=True" in str(err.value) + + +def test_failed_build_output_with_surface_server_logs(client, tmp_path): + env = _env_dir(tmp_path) + image = BoxdProvider.image_from_dockerfile(str(env / "server" / "Dockerfile")) + client.machines.build_exit = 1 + with pytest.raises(RuntimeError, match="build broke"): + _provider(client, surface_server_logs=True).start_container(image) + + +def test_unregistered_dockerfile_uri(client): + with pytest.raises(ValueError, match="image_from_dockerfile"): + _provider(client).start_container("dockerfile:/nowhere/Dockerfile") + assert client.machines.created == [] + + +def test_real_client_is_built_from_api_key_and_region(): + fake_boxd = types.ModuleType("boxd") + fake_boxd.Boxd = MagicMock(name="Boxd") + with patch.dict(sys.modules, {"boxd": fake_boxd}): + BoxdProvider(api_key="test-key", region="nyc") + fake_boxd.Boxd.assert_called_once_with(api_key="test-key", region="nyc")