Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
51a7722
Add bounded operator coding audits with HF Sandboxes
burtenshaw Nov 10, 2026
85692d7
Add reproducible Hub environment validation pipelines
burtenshaw Nov 10, 2026
c1fc62a
Parse candidate Bash fences and retain raw response hashes
burtenshaw Nov 10, 2026
665c454
Merge candidate response formatting support
burtenshaw Nov 10, 2026
728350d
fix(coding-audit): reject linked artifacts while snapshotting
burtenshaw Nov 10, 2026
b33494f
fix(coding-audit): reject empty tool-call completions
burtenshaw Nov 10, 2026
4278afa
fix(coding-audit): correct offline test import path
burtenshaw Nov 10, 2026
6bea568
fix(hub-validation): gate every paid launch on a bound reservation
burtenshaw Nov 10, 2026
9289701
fix(hub-validation): verify captured tasks against source identities
burtenshaw Nov 10, 2026
3656065
fix(hub-validation): bound source downloads and Harbor archives
burtenshaw Nov 10, 2026
70319a5
Merge #1592 review fixes into #1593
burtenshaw Nov 10, 2026
feb1514
Merge native validation and accepted coding-audit base
burtenshaw Nov 10, 2026
f493218
fix(coding-audit): budget target prompts and permit bounded tool repair
burtenshaw Nov 10, 2026
6f7df1b
fix(hub-validation): honor recipe defaults and persist launch receipts
burtenshaw Nov 10, 2026
a55f117
test(hub-validation): use a generic archive escape target
burtenshaw Nov 10, 2026
e795ea2
fix(coding-audit): retain safe HTTP failure status
burtenshaw Nov 10, 2026
b1fe5e6
fix(coding-audit): declare candidate file-writing contract
burtenshaw Nov 10, 2026
8c60974
test(coding-audit): assert explicit candidate outputs and unchanged t…
burtenshaw Nov 10, 2026
dd58708
Merge commit '7a5af24c1225d93261b6dcbde1fd0b39929c6339' into ben/hub-…
burtenshaw Nov 11, 2026
505edbd
fix(examples): align native admission with validator contract
burtenshaw Nov 12, 2026
73f6a8d
Merge commit '7df6474e5a022cb2279db1d812fad54f9f8ba0b0' into ben/hub-…
burtenshaw Nov 12, 2026
31ef4d3
Merge commit 'd1e748b04e0835bae98b6960abd7a38121fc4e4e' into ben/hub-…
burtenshaw Nov 12, 2026
fe9363b
Merge commit '46301f8a240c909e0d061942b0932b4ae4426f01' into ben/hub-…
burtenshaw Nov 12, 2026
ddabdbc
Merge remote-tracking branch 'origin/main' into ben/hub-validation-ex…
burtenshaw Nov 12, 2026
1c14058
Fix prepared cohort reproducibility and bounded audit evidence
burtenshaw Nov 12, 2026
9c2c0b3
Address cohort code quality review without weakening failure evidence
burtenshaw Nov 12, 2026
e62b343
Merge remote-tracking branch 'origin/main' into ben/hub-validation-ex…
burtenshaw Nov 12, 2026
baab846
Align Hub example documentation with pinned Harbor image support
burtenshaw Nov 12, 2026
00b8614
Merge remote-tracking branch 'origin/main' into ben/hub-validation-ex…
burtenshaw Nov 12, 2026
64b3f57
Merge commit 'b8c0d92db86724d43332b43b3b8ee6f2c03da551' into ben/hub-…
burtenshaw Nov 12, 2026
af0a03e
Merge commit '3cf9ae2ab657944f82c80e4d8f5a1004f4b701c3' into ben/hub-…
burtenshaw Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .claude/docs/REPO_WALKTHROUGH.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,11 @@ ThinkingBox examples: `example_usage.py` is a public-client smoke test, while `e

## Harbor integration

`examples/coding_audit/` owns the supplemental operator coding audit, its standalone
`python -m examples.coding_audit` CLI, and offline tests. It is separate from core
RFC 008 validation. `examples/hub_validation_pipeline/` prepares and submits its
frozen cloud bundles.

`src/openenv/core/harness/capture/` records inference calls independently of a trainer or tokenizer. `src/openenv/harbor/` runs Harbor tasks, reconciles their trajectories, and exposes the shared training contract through clients and the Gradio playground. `envs/harbor_env/` is the installable environment wrapper. `examples/harbor/nemo_shell_profile/` provides the explicitly qualified NeMo shell workflow.

See `docs/source/guides/harbor-provider-qualification.md` for provider and adapter evidence, `rfcs/012-harbor-capture-providers.md` for explicit evaluation/training and session ownership, and `tests/envs/test_harbor*.py` / `test_capture*.py` for deterministic capture regressions.
291 changes: 291 additions & 0 deletions docs/plans/hub-environment-trainability-validation.md

Large diffs are not rendered by default.

203 changes: 203 additions & 0 deletions docs/plans/hub-environment-validation-pipeline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
# Hub discovery, native validation and model execution

The runnable implementation is [the Hub validation example](../../examples/hub_validation_pipeline/README.md).
It acquires immutable Hugging Face or Harbor Hub task sources, invokes OpenEnv's
native validator, admits tasks against an explicit profile, and optionally runs a
configured model on a new corpus containing only admitted tasks. Source packages
and exclusions are retained. This document describes the reusable implementation
and its qualification boundary, without operational corpus or run records.

Native semantic validation uses the `harbor-semantic-v1` contract and schema 2
evidence introduced by [#1591](https://github.com/openenv/OpenEnv/pull/1591).
The [supplemental coding audit](../../examples/hub_validation_pipeline/CODING_AUDIT.md)
is example code in `examples/coding_audit`, with orchestration in
`examples/hub_validation_pipeline`. Neither a bounded native admission nor a
supplemental audit establishes complete RFC 008 coverage or training benefit.

## Pipeline and ownership

| Stage | Responsibility | Retained output |
| --- | --- | --- |
| Discover | Query both hubs, exhaust pagination and freeze candidate identities. | Candidate metadata and source aliases |
| Acquire | Resolve immutable identities and bound selected downloads. | Original task files, inventories and hashes |
| Enumerate | Identify task roots and preserve ordered canonical identities. | Per-task source records |
| Validate | Use OpenEnv's parser, native collector and standard graders. | JSON reports, raw evidence and checksums |
| Admit | Verify required PASS results and provenance for the selected profile. | Accepted/excluded records with reasons |
| Materialize | Copy admitted tasks with explicit image/resource overlays. | Frozen corpus and index-to-task mapping |
| Evaluate | Use the existing Harbor harness and capture runner. | Every attempted outcome and cleanup evidence |

Core native validation remains in OpenEnv. Discovery, admission policy, cloud
launching, model selection, ratebooks and research experiments remain in examples.
A sibling sandbox or outer `harbor_env` server cannot establish properties of the
actual task sandbox. Every execution copy must bind to its acquired source and
validated image/configuration.

## Acquisition and HF conventions

HF discovery uses both `library:harbor` and `rl-environment`, checks those tags in
returned metadata, and consumes listing pagination before applying a candidate
limit. Trending rank selects discovery priority; it is not a stable task identity.
HF sources require full commit revisions and Harbor packages require full `sha256:`
references. Resolve a mutable HF alias once and record the resulting commit before
acquisition. Never infer byte identity from matching dataset names or task counts.

The SDK's `snapshot_download`/`hf_hub_download` arguments correspond to
`hf download --repo-type dataset --revision ... --local-dir ...`. The example
adds anonymous public-source reads (`token=False`), explicit selected-file/task
bounds and immutable revision checks. It materializes regular files for Harbor
rather than relying on cache links. Gated, disabled, ambiguous or unsupported
sources are exclusions; they do not silently fall back to another registry.
Harbor's Git source pin is checked because a matching package version alone does
not establish that the required listing API exists.

Check Git blob/content hashes and declared file sizes, reject escaping paths,
symlinks, special files and secret-file inputs, and retain the original acquired
inventory. Parsing/acquisition never executes downloaded task scripts on the
controller host. Source revisions do not pin container bases, package downloads
or external services; those need separate image/runtime evidence.

HF publication uses explicit SDK add operations and retained immutable commit IDs.
It does not synchronize a local folder by deleting remote files. This follows the
`hf upload` convention that deletion is explicit. The cloud workflow additionally
requires an existing private result dataset and a fresh content-derived run claim;
ordinary upload success does not authorize duplicate paid execution.

## Native semantic admission

The default `full` profile requires the complete pinned static/runtime/semantic
inventory. Missing or skipped required coverage prevents full admission.
The explicitly selected `harbor-semantic-v1` profile requires:

- `static.manifest`, `runtime.startup`, `runtime.reward_well_formed` and
`runtime.episode_determinism` to PASS.
- `semantic.oracle_max` and `semantic.floor_gap` to PASS, plus
`semantic.canary_floor` when the operator declares canaries.
- No FAIL or ERROR anywhere in the report; no duplicate, unknown or missing
required result, and execution of all required validation levels.
- Schema 2 native evidence for fresh NOP, oracle and oracle-replay Trials,
followed by the declared canaries, all with the same retained image and inspected
configuration, unchanged task identities and verified cleanup.
- Bound raw reward files and successful oracle exits. Recompute replay equality,
the declared oracle maximum, the NOP gap and canary floors from those files.
- Scale tolerance and margin by `high - low`: require
`oracle >= high - oracle_tolerance * (high - low)` and both `high - floor` and
`oracle - floor` to be at least `floor_margin * (high - low)`.
- Read the retained tag from `runtime.startup.measured.retained_image_tag` and
require equality with checksummed native/trial evidence. The validator retains
this tag only for runs without FAIL, ERROR or interruption.

Other standard checks remain named SKIPs and can leave the underlying report WARN.
The admission envelope records these limits beside the unchanged report; it does
not turn the report into full RFC qualification. Missing required evidence is
incomplete, not PASS, and any confirmed failure keeps FAIL precedence.

The contract lives at `validation/openenv.json` in the validation subject and
explicitly declares the reward key/range, tolerance, floor margin, resource
allocation and optional canary directory. A pipeline-supplied overlay records
both the original source digest and the effective validation digest. The current
native adapter supports self-contained, single-step Linux tasks with one main
Docker service, shared verifier execution and public networking. Unsupported resource,
credential, multi-service and extension contracts stay named SKIPs.
The first trial builds the reviewed Dockerfile or pulls a task-declared
`environment.docker_image` pinned as `name@sha256:<digest>`, recorded as
`source_image`. Later trials and model handoff use the inspected immutable image
kept by the startup-recorded retained tag. Mutable tags and tasks with neither
a Dockerfile nor a pinned image remain unsupported; their startup SKIP excludes
them before model handoff. Acquisition preserves the original bytes. A missing
`solution/solve.sh` still produces `semantic.oracle_max` FAIL.
Supplemental cloud audits remain a separate profile and cannot waive this gate.

```bash
uv run --no-sync openenv validate inputs/task --level semantic --policy v2 \
--json --output outputs/task-report.json

uv run --no-sync python -m examples.hub_validation_pipeline.pipeline \
--contract inputs/native-contract.json --profile harbor-semantic-v1 \
--hf-dataset 'org/repo@FULL_COMMIT_SHA' --tasks-per-dataset 1 \
--output outputs/native-selection
```

Use [the README](../../examples/hub_validation_pipeline/README.md) for installation,
Docker requirements and the synthetic demo. The generic example contract is not
a substitute for reviewing another task's reward/resource/canary declarations.

## Safe model handoff and evidence

Build a new run-owned corpus from admitted tasks only. Record its ordered mapping
from transport index to canonical task ID; filtering must not change identities
or expose excluded siblings. Materialize only the validated immutable image and
explicit resource overlay. A trusted pre-execution Trial hook checks each task
copy, source digest, actual image and inspected container configuration.

Pin the model/harness/backend and sampling, token, step and time limits. Hosted
model IDs and returned fingerprints describe available provider evidence; a
public source revision does not attest the weights served remotely. Keep inference
credentials outside validation workers and task arguments.

Record reward maps, the selected headline metric, execution status, errors,
traces, timing and cleanup. A valid reward of zero is a model outcome. Missing
rewards, failed infrastructure and partial result sets remain ungraded/incomplete.
Reconcile expected and returned task IDs. Do not replace failures with zero or
delete their source/evidence. Clean up exact run-owned containers, networks and
volumes; retained image tags are recorded for explicit later removal.

## Supplemental coding audit

`examples/coding_audit/{scoring.py,hf.py,runner.py,__main__.py}` contains the
operator-run `harbor-coding-audit-v1` workflow. It generates bounded candidate
references, repeats exact submissions against fresh verifier instances, tests
reviewed known-wrong cases, probes limited leakage/stale-state behavior and then
measures repeated target attempts. Cloud and pipeline orchestration import this
example package. The controller command is:

```bash
uv run --no-sync python -m examples.coding_audit \
--task inputs/task --config inputs/audit.json --output outputs/audit-report.json
```

Reference solutions, verifier-private files, known-wrong submissions and their
counterexamples remain operator-controlled evaluation inputs. Real-task recipes
require `--panel PATH`; they contain no built-in benchmark selection or negative
panel. The operator panel declares a pinned HF source, task paths, immutable
execution boundaries and complete audit configuration. Its exact byte snapshot
is read once and SHA256-bound to preparation and bundle provenance. Acquisition
preserves every native task/verifier byte.

```bash
uv run --no-sync python -m examples.hub_validation_pipeline.real_tasks \
--panel "$PANEL_PATH" --output outputs/native-recipes --bundle

uv run --no-sync python -m examples.hub_validation_pipeline.full_run \
--panel "$PANEL_PATH" --hf-census "$HF_CENSUS_PATH" \
--harbor-census "$HARBOR_CENSUS_PATH" --budget-usd "$BUDGET_USD" \
--output outputs/native-plans
```

These commands prepare inputs without paid execution. Operator-selected
`--task NAME` values are validated against the supplied panel. A metadata census
is an inventory, not a deduplicated or runtime-qualified corpus. Evaluation-only
panels never become training data.

Paid native-cohort submission additionally requires a shared initialized ledger,
a reservation identifier and `--submit`. The aggregate estimate reserves the
configured request/token ceilings, controller allowance and each child's remote
maximum lifetime before the first submission. The fixed ratebook is a declared
modeling assumption; it is neither a current-price guarantee nor an account-enforced
billing cap. Every initial budget-bearing plan parse and final submission checks
the same expected plan-byte SHA256. Partial or uncertain receipts and unresolved
reservations remain durable for manual reconciliation; launchers never blindly
resume or retry.

## Qualification and research boundary

Keep source-level tests, Linux Docker evidence and live-provider qualification
separate. Framework fakes exercise binding, parsing, incomplete outcomes, budget
races and cleanup accounting; they do not establish a provider's execution or
billing behavior. Supplemental reports always retain `rfc008_complete: false`
and `trainability_established: false`.

Verifier acceptance/rejection controls measure the reviewed panel, not complete
specification correctness. Repeated attempts measure policy-dependent difficulty;
they do not demonstrate learning. Training value requires a separate controlled
experiment with training-permitted data, sealed held-out task families, matched
budgets and independent evaluation. See [the trainability research plan](hub-environment-trainability-validation.md).
28 changes: 28 additions & 0 deletions examples/coding_audit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ alongside request parameters. Support depends on the selected provider/model.
Candidate generation streams bounded content and reasoning; only a complete
`stop` response with validated usage can produce an executable candidate. A
`length` response remains rejected even when its terminal usage is retained.
The candidate prompt declares the allowed output files and requests a Bash script
that writes them. Its versioned contract is recorded with the model metadata;
standalone source-code fences remain errors.
Recognized Bash/sh Markdown fences are parsed before execution; the raw response
and both raw/script hashes are retained. Other language fences are rejected.

Expand All @@ -54,6 +57,31 @@ and `size`, bound to the frozen task and checksummed evidence. This option does
not enumerate task directories or include verifier, reference or specification
files.


Target requests reserve their prompt as well as completion allowance before dispatch.
The frozen policy uses encoded controller-request bytes plus an 8,192-token framing
allowance. This conservative scheduling assumption can stop early; it is not a
serving-tokenizer measurement or a guarantee about remote billing. Evidence records
the reservation, actual per-response usage, request hashes and dispatch decisions.
Reported input above the reservation, output above the request cap, inconsistent
usage, or cumulative usage above the configured total remains an error.
HTTP failures retain their numeric status without response bodies, URLs or headers.
They are not retried automatically; failed-request billing remains unknown.

After a valid interaction prefix, a pre-dispatch budget stop collects only the
allowlisted files actually saved, then uses the normal fresh verifier. A funded,
validated API response ending in `length` is an explicit truncated target attempt:
its partial calls are discarded and existing files are graded. An initial prompt
that cannot fit remains unmeasured. Generator responses still require complete
`stop` output. No failed API, cleanup or grading operation becomes a synthetic
zero reward.

A bindable target tool request with an unknown function name or invalid JSON
argument content gets bounded tool-error feedback and can be repaired within the
same call and token limits. It never executes an unknown method. Malformed call
identities/types, duplicate IDs in one response and backend failures remain errors;
ambiguous batches are rejected before any command executes.

The HF backend launches paid services when invoked without a supplied test
backend. Use reviewed tasks and distinct execution and inference credentials.
The Hub pipeline's cloud worker packages these production example modules with
Expand Down
Loading
Loading