Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
8570efd
feat(validation): add native Harbor oracle-floor pilot
burtenshaw Nov 9, 2026
baefcba
feat(validation): name every unimplemented local check in reports
burtenshaw Nov 10, 2026
2599cf2
feat(validation): grade Harbor tasks with RFC 008 semantic checks
burtenshaw Nov 10, 2026
5e5c342
feat(harbor): let trusted callers hook a trial before it runs
burtenshaw Nov 10, 2026
04ace6d
docs(validation): document the Harbor task binding and remaining RFC …
burtenshaw Nov 10, 2026
abf12d0
fix(validation): handle Harbor validation edge cases
cursoragent Nov 10, 2026
015f3aa
fix(validation): reject Harbor contract numbers that change as floats
burtenshaw Nov 10, 2026
e328ab3
fix(validation): keep the harbor extra unpinned and trial cleanup strict
burtenshaw Nov 10, 2026
7ce0f91
test(validation): install Harbor in the runtime reference project
burtenshaw Nov 10, 2026
156e411
fix(validation): keep Harbor out of fixture export
cursoragent Nov 10, 2026
3148709
test(validation): expect named static SKIPs in the Docker acceptance …
burtenshaw Nov 10, 2026
7109902
test(validation): keep the pop out of an assert
burtenshaw Nov 10, 2026
6d05868
fix(validation): refuse task-declared Harbor images
burtenshaw Nov 10, 2026
e0a901f
fix(validation): fail missing Harbor oracles before execution gates
burtenshaw Nov 10, 2026
47fcf5f
fix(validation): classify Harbor package failures by phase
burtenshaw Nov 10, 2026
51af19c
fix(validation): preserve completed Harbor semantic evidence
burtenshaw Nov 10, 2026
964ac4b
fix(validation): prioritize retained Harbor image failures
burtenshaw Nov 10, 2026
23338b3
fix(validation): preserve resolved targets in Harbor reports
burtenshaw Nov 10, 2026
3d0791e
fix(validation): supervise Harbor builds outside episode budgets
burtenshaw Nov 10, 2026
5530a5d
Merge remote-tracking branch 'origin/main' into ben/rfc008-validation…
burtenshaw Nov 10, 2026
6fdc8ef
fix(validation): cover Harbor host inspection guards
burtenshaw Nov 10, 2026
efe49fd
fix(validation): preserve Harbor inventory failures and safe reasons
burtenshaw Nov 10, 2026
50a4a27
examples: add bounded coding audits with HF Sandboxes (#1592)
burtenshaw Nov 10, 2026
1a165f3
fix(validation): preserve Harbor verifier timeout
cursoragent Nov 10, 2026
67fffc6
Merge Harbor binding review fixes onto 1a165f35
burtenshaw Nov 11, 2026
751eaf7
fix(validation): reject strings in numeric Harbor declarations
burtenshaw Nov 11, 2026
63ba6d1
test(validation): cover Harbor host failures over malformed rewards
burtenshaw Nov 11, 2026
3b24b79
fix(validation): keep the retained Harbor image only for unfailed runs
burtenshaw Nov 11, 2026
e34c0f9
fix(validation): reject Harbor execution in served manifests
burtenshaw Nov 11, 2026
802b389
fix(validation): grade hostile Harbor contract JSON as invalid
burtenshaw Nov 11, 2026
b98a3ac
fix(validation): report Harbor version mismatch as validator error
burtenshaw Nov 11, 2026
b3db157
docs(validation): document a complete pinned Harbor contract
burtenshaw Nov 11, 2026
913e632
fix(validation): normalize semantic thresholds by reward range
burtenshaw Nov 11, 2026
8178528
fix(validation): close rejected trial logs and fail missing evidence
burtenshaw Nov 11, 2026
d617a49
docs(validation): reconcile Harbor semantic and image contracts
burtenshaw Nov 11, 2026
0078387
Merge #1591 contract, grader and RFC review fixes
burtenshaw Nov 11, 2026
7a5af24
chore(validation): sync generated schema descriptions
burtenshaw Nov 11, 2026
b5683b1
fix(validation): remove run-owned Compose build tags
burtenshaw Nov 12, 2026
5cfa06a
fix(validation): preserve errors during image cleanup
burtenshaw Nov 12, 2026
c03bbe9
fix(validation): grade Harbor supervisor timeouts as errors
burtenshaw Nov 12, 2026
bf87446
test(validation): exercise Harbor startup budgets and phase errors
burtenshaw Nov 12, 2026
7df6474
docs(validation): specify Harbor limits and cleanup outcomes
burtenshaw Nov 12, 2026
d1e748b
Merge main after the 0.13.0 release
burtenshaw Nov 12, 2026
082b2f2
Merge main after the 0.13.1.dev0 version bump
burtenshaw Nov 12, 2026
46301f8
Honor coding audit deadline during sandbox readiness
cursoragent Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
316 changes: 316 additions & 0 deletions docs/plans/rfc008-validation-completion.md

Large diffs are not rendered by default.

77 changes: 75 additions & 2 deletions docs/source/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ openenv validate ./my_env --level static
openenv validate ./my_env --level runtime --local --output report.json
```

Runtime validation requires Docker and a `validation.execution` declaration in
For served packages, runtime validation requires Docker and a `validation.execution` declaration in
`openenv.yaml`. The declaration points to a bounded JSON replay plan containing
one reset and a sequence of actions. The validator builds an immutable image,
collects the original episode, repeats the plan in a fresh WebSocket session and
Expand All @@ -69,6 +69,78 @@ supports that capability. Docker validation uses two containers concurrently
during the fresh-container replay. Both use the same immutable image and are
removed even if a check fails.

Harbor `task.toml` directories are also recognized. Validation requires Python 3.12 or newer
and `harbor==0.25.0` installed alongside `openenv[harbor]`:

```bash
pip install "openenv[harbor]" "harbor==0.25.0"
```

A missing or mismatched Harbor installation exits 3 and names `harbor==0.25.0`.

Add a `validation/openenv.json` contract with `profile: "harbor-semantic-v1"`,
`reward_key`, `reward`, `resources` and `types`. The reward range, oracle tolerance
and floor margin must be explicit. Oracle tolerance and floor margin are fractions
of the declared reward range. Numeric strings in numeric fields, oversized integers
and excessive JSON nesting fail `static.manifest`. Reward and resource numbers must
retain their exact value when stored as floats. `resources.gpu_types` remains a list
of strings. This is a minimal complete contract:

```json
{
"profile": "harbor-semantic-v1",
"reward_key": "reward",
"reward": {
"range": [0, 1],
"oracle_tolerance": 0,
"floor_margin": 0.1
},
"resources": {
"cpu": 1,
"memory_mb": 512,
"disk_mb": 1024,
"episode_timeout_s": 300
},
"types": {"tags": ["swe"]}
}
```

The `harbor-semantic-v1` binding runs
fresh Harbor trials in Docker: a do-nothing agent, the oracle (`solution/solve.sh`),
an oracle replay and any canaries declared in the contract. A canary is a known-wrong
`solve.sh` replacement. The results report under `runtime.startup`,
`runtime.reward_well_formed`, `runtime.episode_determinism`, `semantic.oracle_max`,
`semantic.floor_gap` and `semantic.canary_floor`, and `harbor-evidence.json` keeps
the raw trial evidence in the report bundle. Other checks are named SKIPs, so a
passing task still reports WARN. This validates task packages, not the outer
`harbor_env` server.

Supported tasks use Linux, a single step, one Dockerfile, a shared verifier and
public networking. Unsupported shapes include multi-step or non-Linux tasks,
task-declared `environment.docker_image`, missing Dockerfiles, task-authored Compose
files and separate verifier environments. GPU/TPU or external MCP requirements,
credentials, non-public or differing phase networks, task environment variables,
healthchecks, skills, verifier collection and artifact extensions are also
unsupported. A supplied provider, including `--space`, is unsupported. Each case
gives `runtime.startup` SKIP with a named reason and runs no trials. With valid
static declarations and an oracle, the verdict is WARN and the exit code is 0.
A missing `solution/solve.sh` still FAILs `semantic.oracle_max` at the semantic level,
including with `--skip-build` or `--space`.

The initial image build and environment start share a separate 600-second validator
budget. Exceeding it gives `runtime.startup` ERROR. Later environment starts use the
episode budget. There is no shared outer trial deadline. Harbor bounds setup, agent
and verifier separately with the declared `resources.episode_timeout_s`. A timeout
in one of these phases is a package FAIL that names the phase.

The do-nothing trial builds from the task's Dockerfile. The validator keeps
`openenv-validation:<run id>` only when the run has no FAIL or ERROR and was not
interrupted. It records the tag in `runtime.startup.measured` as
`retained_image_tag`. Otherwise it removes the tag. The run-owned Compose build tag
is removed on every run. Tag removal verifies image identity and uses `--no-prune`.
A removal failure marks cleanup incomplete and gives a startup ERROR. During an
abort, cleanup preserves the original exception and records the removal failure.

The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool
Expand All @@ -77,7 +149,8 @@ with unsupported capabilities or incomplete evidence appear as `SKIP`; they make
the result `WARN`, which exits zero and does not mean Level 2 is complete.
`FAIL` exits 1, unsupported package formats exit 2, and internal or policy errors
exit 3. `--level semantic`
includes the available lower-level checks but does not claim semantic execution.
runs the semantic checks for Harbor tasks. Served packages have no oracle execution
yet, so their semantic checks are named SKIPs.
`--skip-build` runs declaration checks and skips runtime execution entirely.
The Docker provider supports CPU workloads in `public` and `no-network` modes;
`allowlist` and GPU requirements skip runtime before building. A `no-network`
Expand Down
66 changes: 66 additions & 0 deletions examples/coding_audit/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Operator coding audit

This example collects and scores bounded coding-task evidence. It belongs to the
operator workflow, separate from the installed OpenEnv validator and its RFC 008
checks. Acceptance here does not establish full RFC 008 conformance, containment,
served-model revision attestation, or training benefit.

From a repository checkout, install the tested SDK and run the example:

```bash
uv sync --group dev
uv pip install 'huggingface_hub==1.33.0'
PYTHONPATH=src:envs uv run --no-sync python -m examples.coding_audit \
--task /path/to/task \
--config examples/coding_audit/config.json \
--output /path/to/new-audit-directory \
--json
```

The output directory is required and must be new. It retains `report.json`,
`evidence.json`, `config.json`, raw evidence artifacts, and `SHA256SUMS`. Exit codes
are `0` for accepted evidence, `1` for rejected or incomplete evidence, and `2` for
configuration or input failure. `--json` prints the report as well.

`scoring.py` provides pure `assess_coding_audit` and `validate_audit_config`
functions. `runner.py` provides `normalize_coding_config`, `coding_budget`,
`validate_coding_task`, and `run_coding_validation`. `hf.py` contains
`HFTaskBackend`, the bounded Sandbox and Inference Providers implementation.
The schemas bind task, verifier, immutable image references and independent execution identities.

Models may declare an explicit boolean `enable_thinking`, forwarded through the
HF client's `extra_body.chat_template_kwargs`. It is omitted by default. A
candidate-only override leaves the target protocol unchanged and is recorded
alongside request parameters. Support depends on the selected provider/model.
Candidate generation streams bounded content and reasoning; only a complete
`stop` response with validated usage can produce an executable candidate. A
`length` response remains rejected even when its terminal usage is retained.
Recognized Bash/sh Markdown fences are parsed before execution; the raw response
and both raw/script hashes are retained. Other language fences are rejected.

Candidate generation can opt into public task API context by adding
`"context_paths": ["environment/source/package/api.py"]` to the `generator`
model configuration. Only those declared regular UTF-8 files are read; paths must
be unique, normalized, and below `environment/source/`. Up to 32 files and 65,536
combined bytes are allowed. Linked files or parents, hidden or private paths,
NUL bytes, invalid UTF-8 and oversized context are rejected before inference.
The existing message and token budgets still apply. No source files are added by
default, and `target.context_paths` is rejected.

The candidate prompt labels each source path and its runtime filename relative to
the configured workdir, with `environment/source/` removed. Each generation record
retains `candidate_context` entries containing `path`, `runtime_path`, `sha256`,
and `size`, bound to the frozen task and checksummed evidence. This option does
not enumerate task directories or include verifier, reference or specification
files.

The HF backend launches paid services when invoked without a supplied test
backend. Use reviewed tasks and distinct execution and inference credentials.
The Hub pipeline's cloud worker packages these production example modules with
its frozen source and task bundle; its tests stay outside the bundle.

Run the offline suite with:

```bash
PYTHONPATH=.:src:envs uv run pytest examples/coding_audit -q
```
1 change: 1 addition & 0 deletions examples/coding_audit/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Operator-owned coding audits, separate from core RFC 008 validation."""
63 changes: 63 additions & 0 deletions examples/coding_audit/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
"""Run a bounded operator coding audit and retain its evidence bundle."""

import argparse
import signal
import sys
import threading
from contextlib import contextmanager
from pathlib import Path

from openenv.validation.runtime.json_values import dumps, loads

from .runner import run_coding_validation


@contextmanager
def _sigterm_as_interrupt():
if threading.current_thread() is not threading.main_thread():
yield
return

def interrupt(signum, frame):
raise KeyboardInterrupt

previous = signal.signal(signal.SIGTERM, interrupt)
try:
yield
finally:
signal.signal(signal.SIGTERM, previous)


def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--task", type=Path, required=True)
parser.add_argument("--config", type=Path, required=True)
parser.add_argument(
"--output",
type=Path,
required=True,
help="New directory for the report, evidence and checksummed artifacts",
)
parser.add_argument("--json", action="store_true", dest="json_output")
args = parser.parse_args(argv)
try:
config = loads(args.config.read_text())
with _sigterm_as_interrupt():
report = run_coding_validation(args.task, config, args.output)
except (OSError, ValueError) as error:
print(
f"Coding audit configuration/input failure ({type(error).__name__})",
file=sys.stderr,
)
return 2
if args.json_output:
print(dumps(report))
else:
print(f"Coding audit: {report['verdict'].upper()} ({report['profile']})")
for name, check in report["checks"].items():
print(f" {check['status'].upper():5s} {name}")
return 0 if report["accepted"] else 1


if __name__ == "__main__":
raise SystemExit(main())
83 changes: 83 additions & 0 deletions examples/coding_audit/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
{
"schema_version": 1,
"profile": "harbor-coding-audit-v1",
"target": {
"model_id": "Qwen/Qwen3.8-27B",
"revision": "37561816e66b5b9153b1b64c10968c18c1e6f4a0",
"provider": "deepinfra",
"supports_seed": false,
"supports_tools": true,
"temperature": 0.7,
"max_steps": 16,
"max_tokens": 2048,
"max_total_tokens": 16384
},
"execution": {
"agent_image": "docker.io/library/python@sha256:48d884813cf44e927cf64c5529dd630dd3b18338c6dc727991eef9b71e934a20",
"verifier_image": "docker.io/library/python@sha256:48d884813cf44e927cf64c5529dd630dd3b18338c6dc727991eef9b71e934a20",
"flavor": "cpu-basic",
"timeout_seconds": 120,
"workdir": "/app",
"submission_paths": [
"/app/answer.txt"
],
"private_paths": [
"/tests",
"/solution"
],
"verifier_command": [
"/bin/bash",
"/tests/test.sh"
],
"reward_path": "/logs/verifier/reward.txt",
"reward_key": "reward"
},
"validation": {
"reference_repeats": 3,
"target_attempts": 8,
"reward_min": 0,
"reward_max": 1,
"oracle_tolerance": 0,
"floor_margin": 0.1,
"wrong_answer_max": 0,
"required_negative_cases": [
"wrong-value",
"empty-answer",
"near-miss"
]
},
"generation": {
"candidate_count": 2
},
"negative_cases": [
{
"id": "wrong-value",
"script": "mkdir -p /app; printf '0\\n' > /app/answer.txt"
},
{
"id": "empty-answer",
"script": "mkdir -p /app; : > /app/answer.txt"
},
{
"id": "near-miss",
"script": "mkdir -p /app; printf '41\\n' > /app/answer.txt"
}
],
"seeds": [
1001,
1002,
1003,
1004,
1005,
1006,
1007,
1008
],
"cloud": {
"results_repo": "example-org/private-audit-results",
"flavor": "cpu-basic",
"timeout_seconds": 1800,
"max_sandboxes": 64,
"max_inference_calls": 160
}
}
Loading
Loading