Skip to content
Merged
16 changes: 11 additions & 5 deletions docs/plans/rfc008-validation-completion.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ Three limits apply across bindings:
- **Providers.** Docker-local enforces `public` and `no-network` networking and
refuses allowlists and GPUs. The HF Sandbox provider refuses every launch with
`missing provider capability: immutable_image`, because nothing yet binds a Space
revision to an image digest.
revision to an image digest. Harbor tasks with a digest-pinned image can run
their trials on HF Jobs with `--sandbox hf-sandbox`.
- **Served oracles.** RFC 008 does not yet say how to execute an oracle against a
served WebSocket environment, so served packages report their semantic checks as
named SKIPs.
Expand Down Expand Up @@ -92,7 +93,7 @@ or GPU time. Owners are suggestions for discussion.
| 3 | Verifier strength | #28, #26 procedure | Harbor canaries done; panel open | M, then L | Validation core and Hub pipeline |
| 4 | Determinism, replay and leakage graders | #30, #39, #40, #44 | Open | M | Validation core |
| 5 | Static graders and complete reports | #1, #3, #4, #8, #9, #10, #12, #41, #42 | Reports done; graders open | S to M | Any contributor |
| 6 | Cloud execution on HF Sandbox | Provider qualification | Open | M | Hub pipeline |
| 6 | Cloud execution on HF Sandbox | Provider qualification | Harbor done; Spaces open | M | Hub pipeline |
| 7 | Statistical reference procedure | #13 to #16, #27 | Open | L | Hub pipeline |
| 8 | Wider task shapes | Workspace handoff, multi-container, credentials | Open | M to L | Hub pipeline and validation core |
| 9 | Training signal | #17 | Open | L | Validation experiments and RFC 013 |
Expand Down Expand Up @@ -208,9 +209,14 @@ contributor; these make good first issues.
### 6. Cloud execution on HF Sandbox

The HF Sandbox provider refuses every launch because it has no immutable image
binding. Runtime and semantic checks therefore cannot run in the cloud, or on GPUs,
through the RFC provider interface. Cloud audits work around this with their own
runner.
binding. Runtime and semantic checks for served Spaces therefore cannot run in the
cloud, or on GPUs, through the RFC provider interface.

Harbor tasks do not have this gap when they pin `environment.docker_image` by
digest, because the task declares its own immutable image. `openenv validate
--sandbox hf-sandbox` now runs their trials as HF Jobs through Harbor's
`hf-sandbox` environment, so cloud audits can call `openenv validate` instead of
their own runner. The remaining work below covers served Spaces.

**Approach.** Resolve a Space revision to the registry digest of the image it
builds, verify it, and launch by digest. Qualify public Space images first, then add
Expand Down
37 changes: 35 additions & 2 deletions docs/source/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,8 @@ of strings. This is a minimal complete contract:
```

The `harbor-semantic-v1` binding runs
fresh Harbor trials in Docker: a do-nothing agent, the oracle (`solution/solve.sh`),
fresh Harbor trials, in local Docker unless `--sandbox hf-sandbox` moves them to
HF Jobs: a do-nothing agent, the oracle (`solution/solve.sh`),
an oracle replay and any canaries declared in the contract. A canary is a known-wrong
`solve.sh` replacement. The results report under `runtime.startup`,
`runtime.reward_well_formed`, `runtime.episode_determinism`, `semantic.oracle_max`,
Expand All @@ -126,7 +127,7 @@ Dockerfile nor a pinned image, task-authored Compose files and separate verifier
environments. GPU/TPU or external MCP requirements,
credentials, non-public or differing phase networks, task environment variables,
healthchecks, skills, verifier collection and artifact extensions are also
unsupported. A supplied provider, including `--space`, is unsupported. Each case
unsupported. A served-package provider, including `--space`, is unsupported. Each case
gives `runtime.startup` SKIP with a named reason and runs no trials. With valid
static declarations and an oracle, the verdict is WARN and the exit code is 0.
A missing `solution/solve.sh` still FAILs `semantic.oracle_max` at the semantic level,
Expand All @@ -146,6 +147,38 @@ is removed on every run. Tag removal verifies image identity and uses `--no-prun
A removal failure marks cleanup incomplete and gives a startup ERROR. During an
abort, cleanup preserves the original exception and records the removal failure.

`--sandbox hf-sandbox` runs the same trials on Hugging Face Jobs instead, one job
per trial, through Harbor's `hf-sandbox` environment:

```bash
openenv validate path/to/task --sandbox hf-sandbox
openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade --namespace my-org
```

A sandbox cannot build a Dockerfile, so the task must pin a pullable image as
`environment.docker_image = "name@sha256:<digest>"`. Build and push the image
first, then pin its digest. The image needs `/bin/bash`, which Harbor uses to run
commands, and either `sha256sum` or `openssl`, which the sandbox server uses to
verify its own download. With nothing
to build, every trial starts at once. `--flavor` and `--namespace` mean what they
mean for `hf jobs run`: the flavor sets the hardware, and the namespace is the user
or organization that owns and pays for the jobs, by default the token's user. The
flavor must provide the CPU, memory and disk the contract declares, or
`runtime.startup` reports ERROR before any job starts. Tasks that set an agent or
verifier user are unsupported, because the sandbox cannot switch users.

Each job carries `openenv-validation-run` and `openenv-validation-role` labels. At
`AGENT_START` the validator reads the job back and requires the pinned image, the
requested flavor and a `RUNNING` stage. After each trial it cancels any labelled
job that is still running and waits up to 60 seconds for all of them to stop.
Otherwise cleanup is unverified and `runtime.startup` reports ERROR.
`runtime.startup.measured` records the flavor, namespace and job IDs, and
`harbor-evidence.json` states that the image is the digest requested from HF Jobs,
not one measured inside the sandbox. Check IDs, graders and thresholds are the
same as on Docker, and no image is retained. A served package gives a named
`runtime.startup` SKIP with `--sandbox hf-sandbox`; validate a served Space with
`--space` instead.

The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool
Expand Down
18 changes: 16 additions & 2 deletions rfcs/008-environment-auto-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -756,8 +756,8 @@ an `environment.docker_image` without a digest, tasks with neither a Dockerfile
a pinned image, task-authored Compose files and separate verifier environments.
GPU/TPU or external MCP requirements, credentials, non-public or differing phase
networks, task environment variables, healthchecks, skills, verifier collection and
artifact extensions are also unsupported. A supplied provider, including `--space`,
is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
artifact extensions are also unsupported. A served-package provider, including
`--space`, is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
no trials. With valid static declarations and an oracle, the verdict is WARN and
the exit code is 0. A missing `solution/solve.sh` still FAILs `semantic.oracle_max`
at the semantic level, including with `--skip-build` or `--space`.
Expand Down Expand Up @@ -792,6 +792,20 @@ containers' actual mounts, and verifies their absence. The oracle's exit code is
read at `AGENT_END`, before sandbox log synchronization can overwrite it, and a
later change fails the run.

With `--sandbox hf-sandbox`, each trial runs as its own Hugging Face Job through
Harbor's `hf-sandbox` environment, on the flavor and in the namespace the operator
names. The task must declare a digest-pinned `environment.docker_image`. Tasks that
only ship a Dockerfile, and tasks that set agent or verifier users, give
`runtime.startup` SKIP. A flavor smaller than the contract's resources gives
`runtime.startup` ERROR before any job starts. All trials start at once from the
pinned image. Every job carries run and role labels. At `AGENT_START` the validator
reads the job back and requires the pinned image, the requested flavor and a
`RUNNING` stage. After each trial it cancels any labelled job still running and
verifies that every one reaches a terminal stage. The image identity is the digest
requested from HF Jobs and read back from the job, not a digest measured inside the
sandbox, and the evidence states this limitation. Check ids, graders and thresholds
are unchanged.

For `harbor-semantic-v1`, one trial is a one-shot episode with no per-step
observations. A fresh oracle replay that reproduces every reward and the oracle
exit code establishes `runtime.episode_determinism`. This rule is specific to the
Expand Down
60 changes: 54 additions & 6 deletions src/openenv/cli/commands/validate.py
Original file line number Diff line number Diff line change
Expand Up @@ -223,9 +223,29 @@ def validate(
str,
typer.Option(
"--flavor",
help="HF Jobs hardware flavor for --space, e.g. cpu-basic or t4-small",
help=(
"HF Jobs hardware flavor for --space or --sandbox hf-sandbox, "
"e.g. cpu-basic or t4-small"
),
),
] = "cpu-basic",
sandbox: Annotated[
str,
typer.Option(
"--sandbox",
help=(
"Where Harbor task trials run: docker (local daemon) or "
"hf-sandbox (one HF Job per trial)"
),
),
] = "docker",
namespace: Annotated[
str | None,
typer.Option(
"--namespace",
help="User or organization that owns the --sandbox hf-sandbox jobs",
),
] = None,
) -> None:
"""
Validate a local package or a running OpenEnv server.
Expand All @@ -236,9 +256,11 @@ def validate(
applies the severity policy, and emits a report.

Native Harbor tasks require the optional Harbor dependency and an explicit
`validation/openenv.json` contract. Fresh Docker trials grade the oracle, the
do-nothing floor and any declared canaries under the semantic check ids. Checks
without an implementation are named SKIPs, so the report is WARN at best.
`validation/openenv.json` contract. Fresh trials grade the oracle, the
do-nothing floor and any declared canaries under the semantic check ids. They
run on the local Docker daemon, or with `--sandbox hf-sandbox` as one Hugging
Face Job per trial from the task's digest-pinned image. Checks without an
implementation are named SKIPs, so the report is WARN at best.

Exit codes: 0 pass/warn · 1 fail · 2 unrecognized/unsupported package · 3
internal or policy error.
Expand All @@ -260,15 +282,38 @@ def validate(

# Validate a Hub Space on HF Jobs GPU hardware
openenv validate --space openenv/echo_env --level runtime --flavor t4-small

# Run a Harbor task's trials as HF Jobs instead of local Docker
openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade
```
"""
if space is not None and (target is not None or url is not None or local):
typer.echo(
"Error: --space cannot be combined with a path, --url or --local", err=True
)
raise typer.Exit(EXIT_FAIL)
if space is None and (revision is not None or flavor != "cpu-basic"):
typer.echo("Error: --revision and --flavor require --space", err=True)
if sandbox not in ("docker", "hf-sandbox"):
typer.echo(
f"Error: unknown sandbox {sandbox!r}; expected docker or hf-sandbox",
err=True,
)
raise typer.Exit(EXIT_FAIL)
hf_sandbox = sandbox == "hf-sandbox"
running_url = url is not None or (target is not None and _looks_like_url(target))
if hf_sandbox and (space is not None or running_url):
typer.echo(
"Error: --sandbox cannot be combined with --space or a running URL",
err=True,
)
raise typer.Exit(EXIT_FAIL)
if space is None and revision is not None:
typer.echo("Error: --revision requires --space", err=True)
raise typer.Exit(EXIT_FAIL)
if space is None and not hf_sandbox and flavor != "cpu-basic":
typer.echo("Error: --flavor requires --space or --sandbox hf-sandbox", err=True)
raise typer.Exit(EXIT_FAIL)
if namespace is not None and not hf_sandbox:
typer.echo("Error: --namespace requires --sandbox hf-sandbox", err=True)
raise typer.Exit(EXIT_FAIL)

runtime_target = url
Expand Down Expand Up @@ -367,6 +412,9 @@ def validate(
report_target=f"spaces/{space}@{revision}"
if space is not None
else None,
sandbox=sandbox,
flavor=flavor,
namespace=namespace,
)
report_json = write_report(validation_report, output)
except (SignatureError, UnsupportedPackageError) as exc:
Expand Down
Loading
Loading