Repository navigation
feat(validation): run Harbor validation trials on HF Sandbox - #1611
Conversation
`openenv validate <task> --sandbox hf-sandbox [--flavor F] [--namespace N]` runs each harbor-semantic-v1 trial (do-nothing, oracle, oracle replay and canaries) as its own Hugging Face Job through Harbor's hf-sandbox environment, instead of on the local Docker daemon. The task must pin environment.docker_image by digest, so nothing is built and every trial starts at once. HFJobsInspector labels each job with the run and trial role, reads it back at AGENT_START (pinned image, requested flavor, RUNNING), and after each trial cancels any labelled job still running and verifies that all of them stop. Check ids, graders and thresholds are unchanged. Docker validation behaves as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drain sandbox creation before teardown, register the actual job handle before setup failure can clear it, and inspect known jobs omitted from listings before verifying cleanup. Reject unsupported sandbox names at the Python API boundary. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Harbor and validator timeout supervisors can both cancel startup. Preserve the pending worker and its eventual job handle through repeated cancellation, then propagate cancellation for owned teardown. Co-Authored-By: OpenAI Codex <noreply@openai.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Flavor check uses wrong memory units
- Converted advertised HF Jobs RAM and disk from GB to MiB using 1024 before comparing them with the contract fields.
Preview (6d1c97182d)
diff --git a/docs/plans/rfc008-validation-completion.md b/docs/plans/rfc008-validation-completion.md
--- a/docs/plans/rfc008-validation-completion.md
+++ b/docs/plans/rfc008-validation-completion.md
@@ -43,7 +43,8 @@ Three limits apply across bindings:
- **Providers.** Docker-local enforces `public` and `no-network` networking and
refuses allowlists and GPUs. The HF Sandbox provider refuses every launch with
`missing provider capability: immutable_image`, because nothing yet binds a Space
- revision to an image digest.
+ revision to an image digest. Harbor tasks with a digest-pinned image can run
+ their trials on HF Jobs with `--sandbox hf-sandbox`.
- **Served oracles.** RFC 008 does not yet say how to execute an oracle against a
served WebSocket environment, so served packages report their semantic checks as
named SKIPs.
@@ -43,7 +43,8 @@ Three limits apply across bindings:
- **Providers.** Docker-local enforces `public` and `no-network` networking and
refuses allowlists and GPUs. The HF Sandbox provider refuses every launch with
`missing provider capability: immutable_image`, because nothing yet binds a Space
- revision to an image digest.
+ revision to an image digest. Harbor tasks with a digest-pinned image can run
+ their trials on HF Jobs with `--sandbox hf-sandbox`.
- **Served oracles.** RFC 008 does not yet say how to execute an oracle against a
served WebSocket environment, so served packages report their semantic checks as
named SKIPs.
@@ -92,7 +93,7 @@ or GPU time. Owners are suggestions for discussion.
| 3 | Verifier strength | #28, #26 procedure | Harbor canaries done; panel open | M, then L | Validation core and Hub pipeline |
| 4 | Determinism, replay and leakage graders | #30, #39, #40, #44 | Open | M | Validation core |
| 5 | Static graders and complete reports | #1, #3, #4, #8, #9, #10, #12, #41, #42 | Reports done; graders open | S to M | Any contributor |
-| 6 | Cloud execution on HF Sandbox | Provider qualification | Open | M | Hub pipeline |
+| 6 | Cloud execution on HF Sandbox | Provider qualification | Harbor done; Spaces open | M | Hub pipeline |
| 7 | Statistical reference procedure | #13 to #16, #27 | Open | L | Hub pipeline |
| 8 | Wider task shapes | Workspace handoff, multi-container, credentials | Open | M to L | Hub pipeline and validation core |
| 9 | Training signal | #17 | Open | L | Validation experiments and RFC 013 |
@@ -92,7 +93,7 @@ or GPU time. Owners are suggestions for discussion.
| 3 | Verifier strength | #28, #26 procedure | Harbor canaries done; panel open | M, then L | Validation core and Hub pipeline |
| 4 | Determinism, replay and leakage graders | #30, #39, #40, #44 | Open | M | Validation core |
| 5 | Static graders and complete reports | #1, #3, #4, #8, #9, #10, #12, #41, #42 | Reports done; graders open | S to M | Any contributor |
-| 6 | Cloud execution on HF Sandbox | Provider qualification | Open | M | Hub pipeline |
+| 6 | Cloud execution on HF Sandbox | Provider qualification | Harbor done; Spaces open | M | Hub pipeline |
| 7 | Statistical reference procedure | #13 to #16, #27 | Open | L | Hub pipeline |
| 8 | Wider task shapes | Workspace handoff, multi-container, credentials | Open | M to L | Hub pipeline and validation core |
| 9 | Training signal | #17 | Open | L | Validation experiments and RFC 013 |
@@ -208,9 +209,14 @@ contributor; these make good first issues.
### 6. Cloud execution on HF Sandbox
The HF Sandbox provider refuses every launch because it has no immutable image
-binding. Runtime and semantic checks therefore cannot run in the cloud, or on GPUs,
-through the RFC provider interface. Cloud audits work around this with their own
-runner.
+binding. Runtime and semantic checks for served Spaces therefore cannot run in the
+cloud, or on GPUs, through the RFC provider interface.
+
+Harbor tasks do not have this gap when they pin `environment.docker_image` by
+digest, because the task declares its own immutable image. `openenv validate
+--sandbox hf-sandbox` now runs their trials as HF Jobs through Harbor's
+`hf-sandbox` environment, so cloud audits can call `openenv validate` instead of
+their own runner. The remaining work below covers served Spaces.
**Approach.** Resolve a Space revision to the registry digest of the image it
builds, verify it, and launch by digest. Qualify public Space images first, then add
@@ -208,9 +209,14 @@ contributor; these make good first issues.
### 6. Cloud execution on HF Sandbox
The HF Sandbox provider refuses every launch because it has no immutable image
-binding. Runtime and semantic checks therefore cannot run in the cloud, or on GPUs,
-through the RFC provider interface. Cloud audits work around this with their own
-runner.
+binding. Runtime and semantic checks for served Spaces therefore cannot run in the
+cloud, or on GPUs, through the RFC provider interface.
+
+Harbor tasks do not have this gap when they pin `environment.docker_image` by
+digest, because the task declares its own immutable image. `openenv validate
+--sandbox hf-sandbox` now runs their trials as HF Jobs through Harbor's
+`hf-sandbox` environment, so cloud audits can call `openenv validate` instead of
+their own runner. The remaining work below covers served Spaces.
**Approach.** Resolve a Space revision to the registry digest of the image it
builds, verify it, and launch by digest. Qualify public Space images first, then add
diff --git a/docs/source/reference/cli.md b/docs/source/reference/cli.md
--- a/docs/source/reference/cli.md
+++ b/docs/source/reference/cli.md
@@ -106,7 +106,8 @@ of strings. This is a minimal complete contract:The harbor-semantic-v1 binding runs
-fresh Harbor trials in Docker: a do-nothing agent, the oracle (solution/solve.sh),
+fresh Harbor trials, in local Docker unless --sandbox hf-sandbox moves them to
+HF Jobs: a do-nothing agent, the oracle (solution/solve.sh),
an oracle replay and any canaries declared in the contract. A canary is a known-wrong
solve.sh replacement. The results report under runtime.startup,
runtime.reward_well_formed, runtime.episode_determinism, semantic.oracle_max,
@@ -106,7 +106,8 @@ of strings. This is a minimal complete contract:
The `harbor-semantic-v1` binding runs
-fresh Harbor trials in Docker: a do-nothing agent, the oracle (`solution/solve.sh`),
+fresh Harbor trials, in local Docker unless `--sandbox hf-sandbox` moves them to
+HF Jobs: a do-nothing agent, the oracle (`solution/solve.sh`),
an oracle replay and any canaries declared in the contract. A canary is a known-wrong
`solve.sh` replacement. The results report under `runtime.startup`,
`runtime.reward_well_formed`, `runtime.episode_determinism`, `semantic.oracle_max`,
@@ -126,7 +127,7 @@ Dockerfile nor a pinned image, task-authored Compose files and separate verifier
environments. GPU/TPU or external MCP requirements,
credentials, non-public or differing phase networks, task environment variables,
healthchecks, skills, verifier collection and artifact extensions are also
-unsupported. A supplied provider, including `--space`, is unsupported. Each case
+unsupported. A served-package provider, including `--space`, is unsupported. Each case
gives `runtime.startup` SKIP with a named reason and runs no trials. With valid
static declarations and an oracle, the verdict is WARN and the exit code is 0.
A missing `solution/solve.sh` still FAILs `semantic.oracle_max` at the semantic level,
@@ -126,7 +127,7 @@ Dockerfile nor a pinned image, task-authored Compose files and separate verifier
environments. GPU/TPU or external MCP requirements,
credentials, non-public or differing phase networks, task environment variables,
healthchecks, skills, verifier collection and artifact extensions are also
-unsupported. A supplied provider, including `--space`, is unsupported. Each case
+unsupported. A served-package provider, including `--space`, is unsupported. Each case
gives `runtime.startup` SKIP with a named reason and runs no trials. With valid
static declarations and an oracle, the verdict is WARN and the exit code is 0.
A missing `solution/solve.sh` still FAILs `semantic.oracle_max` at the semantic level,
@@ -146,6 +147,38 @@ is removed on every run. Tag removal verifies image identity and uses `--no-prun
A removal failure marks cleanup incomplete and gives a startup ERROR. During an
abort, cleanup preserves the original exception and records the removal failure.
+`--sandbox hf-sandbox` runs the same trials on Hugging Face Jobs instead, one job
+per trial, through Harbor's `hf-sandbox` environment:
+
+```bash
+openenv validate path/to/task --sandbox hf-sandbox
+openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade --namespace my-org
+```
+
+A sandbox cannot build a Dockerfile, so the task must pin a pullable image as
+`environment.docker_image = "name@sha256:<digest>"`. Build and push the image
+first, then pin its digest. The image needs `/bin/bash`, which Harbor uses to run
+commands, and either `sha256sum` or `openssl`, which the sandbox server uses to
+verify its own download. With nothing
+to build, every trial starts at once. `--flavor` and `--namespace` mean what they
+mean for `hf jobs run`: the flavor sets the hardware, and the namespace is the user
+or organization that owns and pays for the jobs, by default the token's user. The
+flavor must provide the CPU, memory and disk the contract declares, or
+`runtime.startup` reports ERROR before any job starts. Tasks that set an agent or
+verifier user are unsupported, because the sandbox cannot switch users.
+
+Each job carries `openenv-validation-run` and `openenv-validation-role` labels. At
+`AGENT_START` the validator reads the job back and requires the pinned image, the
+requested flavor and a `RUNNING` stage. After each trial it cancels any labelled
+job that is still running and waits up to 60 seconds for all of them to stop.
+Otherwise cleanup is unverified and `runtime.startup` reports ERROR.
+`runtime.startup.measured` records the flavor, namespace and job IDs, and
+`harbor-evidence.json` states that the image is the digest requested from HF Jobs,
+not one measured inside the sandbox. Check IDs, graders and thresholds are the
+same as on Docker, and no image is retained. A served package gives a named
+`runtime.startup` SKIP with `--sandbox hf-sandbox`; validate a served Space with
+`--space` instead.
+
The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool
@@ -146,6 +147,38 @@ is removed on every run. Tag removal verifies image identity and uses `--no-prun
A removal failure marks cleanup incomplete and gives a startup ERROR. During an
abort, cleanup preserves the original exception and records the removal failure.
+`--sandbox hf-sandbox` runs the same trials on Hugging Face Jobs instead, one job
+per trial, through Harbor's `hf-sandbox` environment:
+
+```bash
+openenv validate path/to/task --sandbox hf-sandbox
+openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade --namespace my-org
+```
+
+A sandbox cannot build a Dockerfile, so the task must pin a pullable image as
+`environment.docker_image = "name@sha256:<digest>"`. Build and push the image
+first, then pin its digest. The image needs `/bin/bash`, which Harbor uses to run
+commands, and either `sha256sum` or `openssl`, which the sandbox server uses to
+verify its own download. With nothing
+to build, every trial starts at once. `--flavor` and `--namespace` mean what they
+mean for `hf jobs run`: the flavor sets the hardware, and the namespace is the user
+or organization that owns and pays for the jobs, by default the token's user. The
+flavor must provide the CPU, memory and disk the contract declares, or
+`runtime.startup` reports ERROR before any job starts. Tasks that set an agent or
+verifier user are unsupported, because the sandbox cannot switch users.
+
+Each job carries `openenv-validation-run` and `openenv-validation-role` labels. At
+`AGENT_START` the validator reads the job back and requires the pinned image, the
+requested flavor and a `RUNNING` stage. After each trial it cancels any labelled
+job that is still running and waits up to 60 seconds for all of them to stop.
+Otherwise cleanup is unverified and `runtime.startup` reports ERROR.
+`runtime.startup.measured` records the flavor, namespace and job IDs, and
+`harbor-evidence.json` states that the image is the digest requested from HF Jobs,
+not one measured inside the sandbox. Check IDs, graders and thresholds are the
+same as on Docker, and no image is retained. A served package gives a named
+`runtime.startup` SKIP with `--sandbox hf-sandbox`; validate a served Space with
+`--space` instead.
+
The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool
diff --git a/rfcs/008-environment-auto-validation.md b/rfcs/008-environment-auto-validation.md
--- a/rfcs/008-environment-auto-validation.md
+++ b/rfcs/008-environment-auto-validation.md
@@ -756,8 +756,8 @@ an `environment.docker_image` without a digest, tasks with neither a Dockerfile
a pinned image, task-authored Compose files and separate verifier environments.
GPU/TPU or external MCP requirements, credentials, non-public or differing phase
networks, task environment variables, healthchecks, skills, verifier collection and
-artifact extensions are also unsupported. A supplied provider, including `--space`,
-is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
+artifact extensions are also unsupported. A served-package provider, including
+`--space`, is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
no trials. With valid static declarations and an oracle, the verdict is WARN and
the exit code is 0. A missing `solution/solve.sh` still FAILs `semantic.oracle_max`
at the semantic level, including with `--skip-build` or `--space`.
@@ -756,8 +756,8 @@ an `environment.docker_image` without a digest, tasks with neither a Dockerfile
a pinned image, task-authored Compose files and separate verifier environments.
GPU/TPU or external MCP requirements, credentials, non-public or differing phase
networks, task environment variables, healthchecks, skills, verifier collection and
-artifact extensions are also unsupported. A supplied provider, including `--space`,
-is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
+artifact extensions are also unsupported. A served-package provider, including
+`--space`, is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
no trials. With valid static declarations and an oracle, the verdict is WARN and
the exit code is 0. A missing `solution/solve.sh` still FAILs `semantic.oracle_max`
at the semantic level, including with `--skip-build` or `--space`.
@@ -792,6 +792,20 @@ containers' actual mounts, and verifies their absence. The oracle's exit code is
read at `AGENT_END`, before sandbox log synchronization can overwrite it, and a
later change fails the run.
+With `--sandbox hf-sandbox`, each trial runs as its own Hugging Face Job through
+Harbor's `hf-sandbox` environment, on the flavor and in the namespace the operator
+names. The task must declare a digest-pinned `environment.docker_image`. Tasks that
+only ship a Dockerfile, and tasks that set agent or verifier users, give
+`runtime.startup` SKIP. A flavor smaller than the contract's resources gives
+`runtime.startup` ERROR before any job starts. All trials start at once from the
+pinned image. Every job carries run and role labels. At `AGENT_START` the validator
+reads the job back and requires the pinned image, the requested flavor and a
+`RUNNING` stage. After each trial it cancels any labelled job still running and
+verifies that every one reaches a terminal stage. The image identity is the digest
+requested from HF Jobs and read back from the job, not a digest measured inside the
+sandbox, and the evidence states this limitation. Check ids, graders and thresholds
+are unchanged.
+
For `harbor-semantic-v1`, one trial is a one-shot episode with no per-step
observations. A fresh oracle replay that reproduces every reward and the oracle
exit code establishes `runtime.episode_determinism`. This rule is specific to the
@@ -792,6 +792,20 @@ containers' actual mounts, and verifies their absence. The oracle's exit code is
read at `AGENT_END`, before sandbox log synchronization can overwrite it, and a
later change fails the run.
+With `--sandbox hf-sandbox`, each trial runs as its own Hugging Face Job through
+Harbor's `hf-sandbox` environment, on the flavor and in the namespace the operator
+names. The task must declare a digest-pinned `environment.docker_image`. Tasks that
+only ship a Dockerfile, and tasks that set agent or verifier users, give
+`runtime.startup` SKIP. A flavor smaller than the contract's resources gives
+`runtime.startup` ERROR before any job starts. All trials start at once from the
+pinned image. Every job carries run and role labels. At `AGENT_START` the validator
+reads the job back and requires the pinned image, the requested flavor and a
+`RUNNING` stage. After each trial it cancels any labelled job still running and
+verifies that every one reaches a terminal stage. The image identity is the digest
+requested from HF Jobs and read back from the job, not a digest measured inside the
+sandbox, and the evidence states this limitation. Check ids, graders and thresholds
+are unchanged.
+
For `harbor-semantic-v1`, one trial is a one-shot episode with no per-step
observations. A fresh oracle replay that reproduces every reward and the oracle
exit code establishes `runtime.episode_determinism`. This rule is specific to the
diff --git a/src/openenv/cli/commands/validate.py b/src/openenv/cli/commands/validate.py
--- a/src/openenv/cli/commands/validate.py
+++ b/src/openenv/cli/commands/validate.py
@@ -223,9 +223,29 @@ def validate(
str,
typer.Option(
"--flavor",
- help="HF Jobs hardware flavor for --space, e.g. cpu-basic or t4-small",
+ help=(
+ "HF Jobs hardware flavor for --space or --sandbox hf-sandbox, "
+ "e.g. cpu-basic or t4-small"
+ ),
),
] = "cpu-basic",
+ sandbox: Annotated[
+ str,
+ typer.Option(
+ "--sandbox",
+ help=(
+ "Where Harbor task trials run: docker (local daemon) or "
+ "hf-sandbox (one HF Job per trial)"
+ ),
+ ),
+ ] = "docker",
+ namespace: Annotated[
+ str | None,
+ typer.Option(
+ "--namespace",
+ help="User or organization that owns the --sandbox hf-sandbox jobs",
+ ),
+ ] = None,
) -> None:
"""
Validate a local package or a running OpenEnv server.
@@ -223,9 +223,29 @@ def validate(
str,
typer.Option(
"--flavor",
- help="HF Jobs hardware flavor for --space, e.g. cpu-basic or t4-small",
+ help=(
+ "HF Jobs hardware flavor for --space or --sandbox hf-sandbox, "
+ "e.g. cpu-basic or t4-small"
+ ),
),
] = "cpu-basic",
+ sandbox: Annotated[
+ str,
+ typer.Option(
+ "--sandbox",
+ help=(
+ "Where Harbor task trials run: docker (local daemon) or "
+ "hf-sandbox (one HF Job per trial)"
+ ),
+ ),
+ ] = "docker",
+ namespace: Annotated[
+ str | None,
+ typer.Option(
+ "--namespace",
+ help="User or organization that owns the --sandbox hf-sandbox jobs",
+ ),
+ ] = None,
) -> None:
"""
Validate a local package or a running OpenEnv server.
@@ -236,9 +256,11 @@ def validate(
applies the severity policy, and emits a report.
Native Harbor tasks require the optional Harbor dependency and an explicit
- `validation/openenv.json` contract. Fresh Docker trials grade the oracle, the
- do-nothing floor and any declared canaries under the semantic check ids. Checks
- without an implementation are named SKIPs, so the report is WARN at best.
+ `validation/openenv.json` contract. Fresh trials grade the oracle, the
+ do-nothing floor and any declared canaries under the semantic check ids. They
+ run on the local Docker daemon, or with `--sandbox hf-sandbox` as one Hugging
+ Face Job per trial from the task's digest-pinned image. Checks without an
+ implementation are named SKIPs, so the report is WARN at best.
Exit codes: 0 pass/warn · 1 fail · 2 unrecognized/unsupported package · 3
internal or policy error.
@@ -236,9 +256,11 @@ def validate(
applies the severity policy, and emits a report.
Native Harbor tasks require the optional Harbor dependency and an explicit
- `validation/openenv.json` contract. Fresh Docker trials grade the oracle, the
- do-nothing floor and any declared canaries under the semantic check ids. Checks
- without an implementation are named SKIPs, so the report is WARN at best.
+ `validation/openenv.json` contract. Fresh trials grade the oracle, the
+ do-nothing floor and any declared canaries under the semantic check ids. They
+ run on the local Docker daemon, or with `--sandbox hf-sandbox` as one Hugging
+ Face Job per trial from the task's digest-pinned image. Checks without an
+ implementation are named SKIPs, so the report is WARN at best.
Exit codes: 0 pass/warn · 1 fail · 2 unrecognized/unsupported package · 3
internal or policy error.
@@ -260,15 +282,38 @@ def validate(
# Validate a Hub Space on HF Jobs GPU hardware
openenv validate --space openenv/echo_env --level runtime --flavor t4-small
+
+ # Run a Harbor task's trials as HF Jobs instead of local Docker
+ openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade
```
"""
if space is not None and (target is not None or url is not None or local):
typer.echo(
"Error: --space cannot be combined with a path, --url or --local", err=True
)
raise typer.Exit(EXIT_FAIL)
- if space is None and (revision is not None or flavor != "cpu-basic"):
- typer.echo("Error: --revision and --flavor require --space", err=True)
+ if sandbox not in ("docker", "hf-sandbox"):
+ typer.echo(
+ f"Error: unknown sandbox {sandbox!r}; expected docker or hf-sandbox",
+ err=True,
+ )
+ raise typer.Exit(EXIT_FAIL)
+ hf_sandbox = sandbox == "hf-sandbox"
+ running_url = url is not None or (target is not None and _looks_like_url(target))
+ if hf_sandbox and (space is not None or running_url):
+ typer.echo(
+ "Error: --sandbox cannot be combined with --space or a running URL",
+ err=True,
+ )
+ raise typer.Exit(EXIT_FAIL)
+ if space is None and revision is not None:
+ typer.echo("Error: --revision requires --space", err=True)
+ raise typer.Exit(EXIT_FAIL)
+ if space is None and not hf_sandbox and flavor != "cpu-basic":
+ typer.echo("Error: --flavor requires --space or --sandbox hf-sandbox", err=True)
+ raise typer.Exit(EXIT_FAIL)
+ if namespace is not None and not hf_sandbox:
+ typer.echo("Error: --namespace requires --sandbox hf-sandbox", err=True)
raise typer.Exit(EXIT_FAIL)
runtime_target = url
@@ -260,15 +282,38 @@ def validate(
# Validate a Hub Space on HF Jobs GPU hardware
openenv validate --space openenv/echo_env --level runtime --flavor t4-small
+
+ # Run a Harbor task's trials as HF Jobs instead of local Docker
+ openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade
```
"""
if space is not None and (target is not None or url is not None or local):
typer.echo(
"Error: --space cannot be combined with a path, --url or --local", err=True
)
raise typer.Exit(EXIT_FAIL)
- if space is None and (revision is not None or flavor != "cpu-basic"):
- typer.echo("Error: --revision and --flavor require --space", err=True)
+ if sandbox not in ("docker", "hf-sandbox"):
+ typer.echo(
+ f"Error: unknown sandbox {sandbox!r}; expected docker or hf-sandbox",
+ err=True,
+ )
+ raise typer.Exit(EXIT_FAIL)
+ hf_sandbox = sandbox == "hf-sandbox"
+ running_url = url is not None or (target is not None and _looks_like_url(target))
+ if hf_sandbox and (space is not None or running_url):
+ typer.echo(
+ "Error: --sandbox cannot be combined with --space or a running URL",
+ err=True,
+ )
+ raise typer.Exit(EXIT_FAIL)
+ if space is None and revision is not None:
+ typer.echo("Error: --revision requires --space", err=True)
+ raise typer.Exit(EXIT_FAIL)
+ if space is None and not hf_sandbox and flavor != "cpu-basic":
+ typer.echo("Error: --flavor requires --space or --sandbox hf-sandbox", err=True)
+ raise typer.Exit(EXIT_FAIL)
+ if namespace is not None and not hf_sandbox:
+ typer.echo("Error: --namespace requires --sandbox hf-sandbox", err=True)
raise typer.Exit(EXIT_FAIL)
runtime_target = url
@@ -367,6 +412,9 @@ def validate(
report_target=f"spaces/{space}@{revision}"
if space is not None
else None,
+ sandbox=sandbox,
+ flavor=flavor,
+ namespace=namespace,
)
report_json = write_report(validation_report, output)
except (SignatureError, UnsupportedPackageError) as exc:
@@ -367,6 +412,9 @@ def validate(
report_target=f"spaces/{space}@{revision}"
if space is not None
else None,
+ sandbox=sandbox,
+ flavor=flavor,
+ namespace=namespace,
)
report_json = write_report(validation_report, output)
except (SignatureError, UnsupportedPackageError) as exc:
diff --git a/src/openenv/validation/harbor.py b/src/openenv/validation/harbor.py
--- a/src/openenv/validation/harbor.py
+++ b/src/openenv/validation/harbor.py
@@ -1,13 +1,15 @@
-"""Native Harbor execution: fresh Docker trials graded through RFC 008 check ids.
+"""Native Harbor execution: fresh trials graded through RFC 008 check ids.
-No served-protocol evidence is synthesized. A do-nothing trial builds the image, or
-pulls the task's digest-pinned one, and the oracle, its replay and any canaries run
-pinned to that image.
+No served-protocol evidence is synthesized. On local Docker, a do-nothing trial
+builds the image, or pulls the task's digest-pinned one, and the oracle, its replay
+and any canaries run pinned to that image. On `hf-sandbox`, every trial starts at
+once from the task's digest-pinned image, each in its own Hugging Face Job.
"""
import asyncio
import hashlib
import json
+import math
import os
import re
import shutil
@@ -1,13 +1,15 @@
-"""Native Harbor execution: fresh Docker trials graded through RFC 008 check ids.
+"""Native Harbor execution: fresh trials graded through RFC 008 check ids.
-No served-protocol evidence is synthesized. A do-nothing trial builds the image, or
-pulls the task's digest-pinned one, and the oracle, its replay and any canaries run
-pinned to that image.
+No served-protocol evidence is synthesized. On local Docker, a do-nothing trial
+builds the image, or pulls the task's digest-pinned one, and the oracle, its replay
+and any canaries run pinned to that image. On `hf-sandbox`, every trial starts at
+once from the task's digest-pinned image, each in its own Hugging Face Job.
"""
import asyncio
import hashlib
import json
+import math
import os
import re
import shutil
@@ -17,6 +19,7 @@
import tempfile
import time
import uuid
+from concurrent.futures import ThreadPoolExecutor
from dataclasses import replace
from decimal import Decimal
from fractions import Fraction
@@ -17,6 +19,7 @@
import tempfile
import time
import uuid
+from concurrent.futures import ThreadPoolExecutor
from dataclasses import replace
from decimal import Decimal
from fractions import Fraction
@@ -58,6 +61,10 @@
"RetainedImageDrift",
"RetainedImageUnverified",
}
+_TERMINAL_STAGES = {"COMPLETED", "CANCELED", "ERROR", "DELETED"}
+_JOB_STOP_TIMEOUT_S = 60
+# Readiness budget inside the 600-second start budget; allocation can take longer.
+_SANDBOX_START_TIMEOUT_S = 540
# Checks this binding measures; every other local check is a named SKIP.
_IMPLEMENTED = {
"runtime.startup",
@@ -58,6 +61,10 @@
"RetainedImageDrift",
"RetainedImageUnverified",
}
+_TERMINAL_STAGES = {"COMPLETED", "CANCELED", "ERROR", "DELETED"}
+_JOB_STOP_TIMEOUT_S = 60
+# Readiness budget inside the 600-second start budget; allocation can take longer.
+_SANDBOX_START_TIMEOUT_S = 540
# Checks this binding measures; every other local check is a named SKIP.
_IMPLEMENTED = {
"runtime.startup",
@@ -276,7 +283,147 @@ def remove_image(self):
self._retained_image = None
-def _unsupported(task, contract):
+def _job_labels(trial_name):
+ # openenv-validation-<run id>-<role>
+ run_id, role = trial_name.split("-", 3)[2:]
+ return {"openenv-validation-run": run_id, "openenv-validation-role": role}
+
+
+class HFJobsInspector:
+ """
+ Trusted HF Jobs checks limited to sandbox jobs labelled with this run.
+
+ The image identity is the digest requested from HF Jobs and read back from the
+ job; no runtime digest is measured inside the sandbox.
+ """
+
+ def __init__(self, image, flavor, namespace, resources):
+ self.image = image
+ self.flavor = flavor
+ self.namespace = namespace
+ self._resources = resources
+ self._api = None
+ self._observed_jobs = {}
+
+ def preflight(self):
+ from huggingface_hub import HfApi
+
+ self._api = HfApi()
+ if self.namespace is None:
+ self.namespace = self._api.whoami()["name"]
+ hardware = {item.name: item for item in self._api.list_jobs_hardware()}
+ if self.flavor not in hardware:
+ raise NativeEvidenceError(f"unknown HF Jobs flavor {self.flavor}")
+ item = hardware[self.flavor]
+ # The Hub lists specs as "2 vCPU", "16 GB" and "50 GB".
+ cpu, memory_gb, disk_gb = (
+ int(spec.split()[0])
+ for spec in (item.cpu, item.ram, item.ephemeral_storage)
+ )
+ if (
+ cpu < self._resources.cpu
+ or memory_gb * 1024 < self._resources.memory_mb
+ or disk_gb * 1024 < self._resources.disk_mb
+ ):
+ raise NativeEvidenceError(
+ f"HF Jobs flavor {self.flavor} ({item.cpu}, {item.ram} RAM, "
+ + f"{item.ephemeral_storage} disk) is smaller than the contract resources"
+ )
+ return {
+ "flavor": self.flavor,
+ "namespace": self.namespace,
+ "hardware": {
+ "cpu": item.cpu,
+ "ram": item.ram,
+ "disk": item.ephemeral_storage,
+ },
+ }
+
+ def register_job(self, project, job_id):
+ # The Sandbox handle is authoritative even before AGENT_START or listing.
+ self._observed_jobs.setdefault(project, set()).add(job_id)
+
+ def _jobs(self, project):
+ if not _PROJECT.fullmatch(project):
+ raise NativeEvidenceError("not a run-owned Harbor project")
+ labels = _job_labels(project.removesuffix("__env"))
+ jobs = list(self._api.list_jobs(labels=labels, namespace=self.namespace))
+ # A listing omission does not prove a previously observed job terminated.
+ listed_ids = {job.id for job in jobs}
+ known_ids = self._observed_jobs.get(project, set())
+ jobs.extend(
+ self._api.inspect_job(job_id=job_id, namespace=self.namespace)
+ for job_id in sorted(known_ids - listed_ids)
+ )
+ # Cleanup cancels what this returns, so recheck the server-side filter.
+ if any(
+ not labels.items() <= (job.labels or {}).items()
+ or job.owner.name != self.namespace
+ for job in jobs
+ ):
+ raise NativeEvidenceError("HF Jobs returned a job outside this run")
+ self._observed_jobs[project] = known_ids | listed_ids
+ return jobs
+
+ def inventory(self, project):
+ return {"jobs": sorted(job.id for job in self._jobs(project))}
+
+ def capture(self, project, retained_tag, *, retain=True):
+ jobs = self._jobs(project)
+ if len(jobs) != 1:
+ raise NativeEvidenceError("HF Sandbox trial requires exactly one job")
+ job = jobs[0]
+ if (
+ job.status.stage != "RUNNING"
+ or job.docker_image != self.image
+ or job.flavor != self.flavor
+ ):
+ raise NativeEvidenceError(
+ "HF Sandbox job is not running the pinned image on the requested flavor"
+ )
+ configuration = {
+ "image_ref": job.docker_image,
+ "flavor": self.flavor,
+ "namespace": self.namespace,
+ }
+ return {
+ "container_id": job.id,
+ "project": project,
+ "image_ref": job.docker_image,
+ "configuration": configuration,
+ "configuration_digest": _digest(configuration),
+ }
+
+ def cleanup(self, project):
+ # Startup settles allocation before teardown; inspect until each job ends.
+ deadline = time.monotonic() + _JOB_STOP_TIMEOUT_S
+ cancelled = set()
+ while True:
+ stages = {job.id: job.status.stage for job in self._jobs(project)}
+ running = {
+ job_id
+ for job_id, stage in stages.items()
+ if stage not in _TERMINAL_STAGES
+ }
+ for job_id in running - cancelled:
+ self._api.cancel_job(job_id=job_id, namespace=self.namespace)
+ cancelled.add(job_id)
+ if not running or time.monotonic() > deadline:
+ return {
+ "verified": not running,
+ "jobs": stages,
+ "cancelled": sorted(cancelled),
+ }
+ time.sleep(2)
+
+ def verify_image(self, tag, image_ref):
+ return image_ref == self.image
+
+ def remove_image(self):
+ pass # Nothing is pulled or tagged on this host.
+
+
+def _unsupported(task, contract, sandbox):
config = task.config
env = config.environment
if task.has_steps:
@@ -276,7 +283,147 @@ def remove_image(self):
self._retained_image = None
-def _unsupported(task, contract):
+def _job_labels(trial_name):
+ # openenv-validation-<run id>-<role>
+ run_id, role = trial_name.split("-", 3)[2:]
+ return {"openenv-validation-run": run_id, "openenv-validation-role": role}
+
+
+class HFJobsInspector:
+ """
+ Trusted HF Jobs checks limited to sandbox jobs labelled with this run.
+
+ The image identity is the digest requested from HF Jobs and read back from the
+ job; no runtime digest is measured inside the sandbox.
+ """
+
+ def __init__(self, image, flavor, namespace, resources):
+ self.image = image
+ self.flavor = flavor
+ self.namespace = namespace
+ self._resources = resources
+ self._api = None
+ self._observed_jobs = {}
+
+ def preflight(self):
+ from huggingface_hub import HfApi
+
+ self._api = HfApi()
+ if self.namespace is None:
+ self.namespace = self._api.whoami()["name"]
+ hardware = {item.name: item for item in self._api.list_jobs_hardware()}
+ if self.flavor not in hardware:
+ raise NativeEvidenceError(f"unknown HF Jobs flavor {self.flavor}")
+ item = hardware[self.flavor]
+ # The Hub lists specs as "2 vCPU", "16 GB" and "50 GB".
+ cpu, memory_gb, disk_gb = (
+ int(spec.split()[0])
+ for spec in (item.cpu, item.ram, item.ephemeral_storage)
+ )
+ if (
+ cpu < self._resources.cpu
+ or memory_gb * 1024 < self._resources.memory_mb
+ or disk_gb * 1024 < self._resources.disk_mb
+ ):
+ raise NativeEvidenceError(
+ f"HF Jobs flavor {self.flavor} ({item.cpu}, {item.ram} RAM, "
+ + f"{item.ephemeral_storage} disk) is smaller than the contract resources"
+ )
+ return {
+ "flavor": self.flavor,
+ "namespace": self.namespace,
+ "hardware": {
+ "cpu": item.cpu,
+ "ram": item.ram,
+ "disk": item.ephemeral_storage,
+ },
+ }
+
+ def register_job(self, project, job_id):
+ # The Sandbox handle is authoritative even before AGENT_START or listing.
+ self._observed_jobs.setdefault(project, set()).add(job_id)
+
+ def _jobs(self, project):
+ if not _PROJECT.fullmatch(project):
+ raise NativeEvidenceError("not a run-owned Harbor project")
+ labels = _job_labels(project.removesuffix("__env"))
+ jobs = list(self._api.list_jobs(labels=labels, namespace=self.namespace))
+ # A listing omission does not prove a previously observed job terminated.
+ listed_ids = {job.id for job in jobs}
+ known_ids = self._observed_jobs.get(project, set())
+ jobs.extend(
+ self._api.inspect_job(job_id=job_id, namespace=self.namespace)
+ for job_id in sorted(known_ids - listed_ids)
+ )
+ # Cleanup cancels what this returns, so recheck the server-side filter.
+ if any(
+ not labels.items() <= (job.labels or {}).items()
+ or job.owner.name != self.namespace
+ for job in jobs
+ ):
+ raise NativeEvidenceError("HF Jobs returned a job outside this run")
+ self._observed_jobs[project] = known_ids | listed_ids
+ return jobs
+
+ def inventory(self, project):
+ return {"jobs": sorted(job.id for job in self._jobs(project))}
+
... diff truncated: showing 800 of 3000 lines
You can send follow-ups to the cloud agent here.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 95050bc. Configure here.
Wait for the cancelled startup task with an explicit deadline while preserving the cancellation assertion and avoiding a CodeQL no-effect finding. Co-Authored-By: OpenAI Codex <noreply@openai.com>
HF Sandbox creation can raise after allocating a job and failing its best-effort cancellation. Record missing startup handles and require recovery of a labelled job before cleanup can be verified. Exercise both recovered and invisible allocations with fakes. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Accept the advertised memory and storage capacities expressed in contract MiB, and reject one MiB above each limit before job allocation. Co-Authored-By: OpenAI Codex <noreply@openai.com>

Summary
openenv validatecan now run a native Harbor task's validation trials on Hugging Face Jobs instead of the local Docker daemon:Until now the
harbor-semantic-v1binding could run only where a Linux Docker daemon was available. It hard-coded Harbor's Docker environment, used the localdockerCLI for preflight, image identity and cleanup, and turned any provider into aruntime.startupSKIP. The one cloud option,--space, covers served Spaces and refuses every launch withimmutable_image.Harbor 0.25.0, which the binding pins, already ships an
hf-sandboxenvironment: each trial runs in a dedicated HF Job throughhuggingface_hub.Sandbox. OpenEnv's Harbor rollout code already offers it, and this PR lets validation use it too. A task that pinsenvironment.docker_imageby digest declares its own immutable image, so theimmutable_imageblocker that stops--spacedoes not apply.Part of RFC 008 (#778). Builds on the Harbor binding (#1591) and digest-pinned task images (#1607).
What changes
--sandbox {docker,hf-sandbox}(defaultdocker) and--namespace.--flavornow also applies to--sandbox hf-sandbox.run_validationgains matchingsandbox,flavorandnamespacekeywords.hf-sandbox, every trial (do-nothing, oracle, oracle replay and each canary) is its own HF Job started from the task's pinned image. Nothing is built, so all trials start at once, on a thread pool sized to the trial count. The Docker path is unchanged.HFJobsInspectortakes the place ofDockerInspectoron this path.openenv-validation-runandopenenv-validation-rolelabels. Harbor's environment does not forward labels or namespace toSandbox.create, so the binding replacesstarton the trial's environment instance, the same per-instance pattern it already uses for Compose.AGENT_START, that job is read back and must run the pinned image on the requested flavor in theRUNNINGstage.list_jobsare checked throughinspect_job; listing absence alone never proves termination. All jobs are rechecked against the labels and namespace before cancellation. Pending creation is drained through cancellation, including overlapping Harbor and validator timeouts, before cleanup runs.runtime.startupSKIP. A flavor smaller than the contract's CPU, memory or disk givesruntime.startupERROR before any job starts. A served package with--sandbox hf-sandboxgets a named SKIP.runtime.startup.measuredaddssandbox,flavor,namespaceandjob_ids.harbor-evidence.jsonaddssandbox,hf_jobsand alimitationsentry, and each trial'scontainer_idholds its job ID. No image is retained on this path.Hugging Face ecosystem
--flavorand--namespacematchhf jobs run --flavor/--namespaceandhuggingface_hub.Sandbox.create(flavor=..., namespace=...), with the same names and defaults (cpu-basic, and the token's user). The valuehf-sandboxmatches Harbor'sEnvironmentType.HF_SANDBOXand the sandbox names in OpenEnv's Harbor rollout code. Jobs are created withSandbox.create, found withHfApi.list_jobs(labels=...)and stopped withHfApi.cancel_job. No new Hub behavior is introduced.Limits
/bin/bash, which Harbor uses to run commands, andsha256sumoropenssl, which the sandbox server uses to check its own download.huggingface_hubSandbox API is marked experimental. The binding already requiresharbor==0.25.0, and thestartreplacement relies on that version'sHFSandboxEnvironment.retained_image_tagor a 64-hexcontainer_id, needs a rule for HF evidence before it can accept these reports.Test plan and results
tests/test_validationandtests/test_cli, 2,218 passed, 66 skipped.95050bc63: installed-wheel fast suite, 1,872 passed with zero skips under--require-complete; every evidence checksum verifies. The recorded source diff matches that commit, and the installed Harbor binding and runner matched those sources by SHA-256.ValueErrorbefore any validation or allocation.timeoutcommand on this macOS host; the focused and pinned suites above completed successfully.startagainst Harbor's realHFSandboxEnvironmentwith a fakeSandbox.create.Live qualification on HF Jobs
The four-task smoke set from #1591, each task pinned by digest to a public
pythonimage, run with--sandbox hf-sandbox --flavor cpu-basic:semantic.floor_gap: the do-nothing reward is 1.0semantic.canary_floor: the wrong answer earns 1.0solution/solve.shsemantic.oracle_max, with no jobs startedThe three runs that started jobs created 12 labelled jobs. All 12 ended
CANCELED, and every evidence bundle's checksums verify. Harbor's own teardown stopped every job, so cleanup never had to cancel one. The correct-verifier task was rerun on the original feature head with the same result in 46 s. The review fixes were validated with fake-boundary and installed-wheel tests; no additional paid jobs were launched.🤖 Generated with Claude Code