Repository navigation
examples: discover, validate and evaluate pinned Hub environments - #1593
Conversation
There was a problem hiding this comment.
Alignment Review Report
Automated Checks
- Lint: PASS —
ruff format --check,ruff check, andusort checkall pass cleanly on every new/changed file underexamples/hub_validation_pipeline/(17 files)..claude/hooks/lint.shitself exits non‑zero, but only on the same ~47 pre‑existingenvs//src/formatting drifts noted inAGENTS.md(e.g.tests/envs/test_grid_world.py,tests/envs/test_julia_env.py) — none touched by this diff, which doesn't modifysrc/,tests/, orenvs/at all. - Debug code: CLEAN —
.claude/hooks/check-debug.shonly scanssrc/, so I also scanned the new files directly: noTODO/FIXME/HACK, nobreakpoint()/pdb, and everyprint(is in an intentional CLI/demo entry point (cloud.py,spend.py,pipeline.py,demo.py,real_tasks.py,full_run.py), not leftover debugging.
Note: this PR is stacked on ben/coding-audit-operator, not main, so Tests/Lint/Package CI (which only trigger on PRs into main/release) won't run here — only build_pr_documentation and the base-filter-free Runtime validation / Linux Docker workflow do, and that workflow exercises tests/validation_runtime/, not this example. The finding below is therefore invisible on this PR's own checks and will only surface once the stack retargets main.
Open RFCs Context
- RFC 008 — Environment Auto-Validation (
rfcs/008-environment-auto-validation.md, Status: In Review, @zkwentz) is directly implicated: this PR'sREADME.md/CODING_AUDIT.mdexplicitly position themselves against it ("Core native validation remains in OpenEnv... Neither corpus admission nor a supplemental coding-audit PASS establishes full RFC 008 qualification"). See Tier 2 below. - Other In‑Review/Draft RFCs (000, 001, 002, 003, 005, 006, 011 Draft, 012, 014) are not materially touched by this change; no conflicts found there.
Tier 1: Fixes Required
-
examples/hub_validation_pipeline/test_sources.py(test_harbor_native_cli_json_and_model_contract,test_harbor_native_acquisition_models) — these usepytest.importorskip("harbor")to gate on the Harbor-git-pin capability, but that guard doesn't actually detect the capability:harbor==0.25.0is already a core dependency (pyproject.toml:75,>=0.22.0) thatuv sync --all-extrasinstalls from PyPI, soimport harborsucceeds and the test runs instead of skipping. It then hitssources._harbor_api()'sRuntimeError("Harbor source capability mismatch...")and fails rather than skipping. Verified directly: running the README's own documented verification command out of the box —givesPYTHONPATH=.:src:envs uv run --no-sync python -m pytest examples/hub_validation_pipeline/ tests/test_validation/test_harbor_validation.py tests/envs/test_harbor_validated_trial_hook.py -q2 failed, 414 passedin the standard repo setup fromAGENTS.md(uv sync --all-extras, no manual pinned-Harbor install). Since this PR is stacked off-main,TestsCI won't catch this either. Fix: skip (not fail) when the installed Harbor lacks the pinned Git capability, e.g. wrap the_harbor_api()capability probe in the test and callpytest.skip(...)onRuntimeError, or check the installed revision directly before calling the real API intest_harbor_native_cli_json_and_model_contract/test_harbor_native_acquisition_models.
No other mechanical issues found: no uninitialized variables, no missing imports/syntax errors, and no credential-exposure issues — cloud.py/real_tasks.py/spend.py are unusually careful here (redaction in main()'s exception handler, _no_secrets() scanning every persisted artifact for hf_... tokens and private-key headers, distinct-secret-value checks in _runtime_secrets()).
Tier 2: Alignment Discussion
Principle Conflicts
ALIGNMENT FLAG: ~8,000 lines of new "example" infrastructure vs. the repo's leanness principle
- Principle/RFC at stake:
CLAUDE.md"Simplicity" ("OpenEnv values leanness and simplicity... Do not add layers of indirection... Do not add defensive code, fallback paths or configuration options 'just in case'... When in doubt, prefer less code.") - The concern: This PR adds 21 files / ~8,079 lines of implementation+test code under
examples/hub_validation_pipeline/(cloud.pyalone is 1,064 lines) implementing a content-addressed bundle format, a custom ZIP-manifest validator, an offline shared-spend ledger with file locking and exact-rational accounting (spend.py), a census/cohort submission pipeline (full_run.py), and bounded-concurrency job polling (prepared_batch.py). This is architecturally far more than an illustrative "example" — it reads like production infrastructure that happens to live outsidesrc/. Worth a maintainer gut-check on whether this scope belongs inexamples/as-is, or should be trimmed/simplified, independent of the RFC question below. - Suggested reviewer: @Darktex (author of
PRINCIPLES.md/INVARIANTS.md/the Simplicity section)
RFC Conflicts
ALIGNMENT FLAG: Cloud submission/spend/queue machinery built on top of RFC 008 while RFC 008 explicitly says the repo ships none of that
- Principle/RFC at stake: RFC 008 (In Review, @zkwentz), "The central architectural commitment: ship local validation and the contracts that let any team build a hub; do not ship a hub... The repo therefore ships no runner, queue, submission API, or gate policy — only the pieces every operator needs to agree on."
- The concern:
cloud.py(submit/stage_inputs/worker, HF Jobs launch specs, three-secret credential model, claim/receipt protocol),spend.py(a shared budget ledger with reservations/settlement — i.e. a gate policy), andfull_run.py/prepared_batch.py(census-driven cohort submission with bounded concurrency — i.e. a queue/coordinator) are, functionally, exactly the "runner, queue, submission API, gate policy" RFC 008 says operators own and the repo does not ship. The PR's own docs are aware of the tension and thread the needle by keeping it underexamples/rather thansrc/openenv/validation/("Core native validation remains in OpenEnv... cloud orchestration... remain example code"), and by repeatedly disclaimingrfc008_complete: false/trainability_established: false. That mitigates but doesn't resolve the conflict: RFC 008's non-goal reads as scoped to "the repo" (i.e. this GitHub repository), not justsrc/, and a maintained, tested, documented HF-Jobs submission+budget system living in-tree is a de facto reference hub implementation — the exact outcome RFC 008 says to avoid ("a first-pass hub implementation in the shared repo would quietly make one operator's answers the standard"). Separately, per therfc-checkskill's own criteria ("New abstractions or design patterns" → RFC required, not restricted tosrc/core/), the bundle-manifest format, panel schema (openenv-native-evaluation-panel-v1), spend-ledger schema (openenv-shared-spend-v1), and cloud-plan schema (openenv-coding-cloud-v1) introduced here are all new, versioned wire contracts that arguably deserved their own RFC rather than landing directly as example code. - Suggested reviewer: @zkwentz (RFC 008 author, owns the no-hub commitment) — @Darktex for the "examples vs. core" line-drawing question.
Summary
- 1 mechanical issue to fix (Tier 1: Harbor-capability test guard fails instead of skipping; invisible on this PR's own CI since it's stacked off
main) - 1 alignment point for human review (scope/complexity of the new example vs. repo leanness principle)
- 1 RFC conflict to discuss (cloud submission/spend/queue machinery vs. RFC 008's explicit "do not ship a hub" commitment)
Sent by Cursor Automation: Pre-review
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 4 total unresolved issues (including 2 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Root task paths are rejected
- Allowed the explicit root locator and mapped it to the Hub API's root-tree request while retaining path validation for all other task paths.
- ✅ Fixed: Budget check uses exact not reserved
- Changed the budget gate to compare the rounded-up reservation amount that the shared ledger actually reserves.
Or push these changes by commenting:
@cursor push 7c9f901ebb
Preview (7c9f901ebb)
diff --git a/examples/hub_validation_pipeline/full_run.py b/examples/hub_validation_pipeline/full_run.py
--- a/examples/hub_validation_pipeline/full_run.py
+++ b/examples/hub_validation_pipeline/full_run.py
@@ -260,7 +260,7 @@
manifest["stage"] = "budget"
manifest["estimate"] = estimate(configs)
manifest["within_budget"] = (
- Fraction(manifest["estimate"]["reservation_usd_exact"]) <= cap
+ Fraction(manifest["estimate"]["reservation_usd_rounded_up"]) <= cap
)
manifest["status"] = "prepared"
_write(manifest_path, manifest)
diff --git a/examples/hub_validation_pipeline/real_tasks.py b/examples/hub_validation_pipeline/real_tasks.py
--- a/examples/hub_validation_pipeline/real_tasks.py
+++ b/examples/hub_validation_pipeline/real_tasks.py
@@ -13,7 +13,7 @@
import stat
import tempfile
import tomllib
-from pathlib import Path
+from pathlib import Path, PurePosixPath
from huggingface_hub import hf_hub_download, HfApi, RepoFile, RepoFolder
from openenv.validation.runner import source_digest
@@ -54,7 +54,12 @@
if not isinstance(task, dict):
raise ValueError("Panel tasks must be objects")
name = cloud._safe_name(task.get("name", ""))
- task_path = cloud._safe_name(task.get("task_path", ""))
+ raw_task_path = task.get("task_path", "")
+ task_path = (
+ PurePosixPath(".")
+ if raw_task_path == "."
+ else cloud._safe_name(raw_task_path)
+ )
if (
len(name.parts) != 1
or name.as_posix().casefold() in names
@@ -103,7 +108,8 @@
"""Acquire only one pinned native tree, with size and Git-blob checks."""
revision = sources._pin(source["revision"], "hf")
repo_id, task_path = source["repo_id"], task["task_path"]
- cloud._safe_name(task_path)
+ if task_path != ".":
+ cloud._safe_name(task_path)
destination = Path(destination)
if destination.exists() or destination.is_symlink():
raise FileExistsError("Acquisition requires a fresh task directory")
@@ -117,7 +123,7 @@
total = 0
for item in api.list_repo_tree(
repo_id,
- path_in_repo=task_path,
+ path_in_repo=None if task_path == "." else task_path,
repo_type="dataset",
revision=revision,
recursive=True,You can send follow-ups to the cloud agent here.
Read original task files through the descriptor-based reader before writing the frozen copy, so copying cannot erase hard-link evidence. Cover context, instruction, verifier and reference artifacts through the runner with offline SDK fakes.
Require a nonempty tool-call list when the provider finishes with tool_calls. Exercise null, missing and empty calls through collection and scoring after a valid terminal action, and verify failed targets are never graded.
Include the repository root in PYTHONPATH so pytest can import the examples package. Verified the documented command with all 417 offline tests passing.
Remove the unguarded CLI and discovery submission paths. Bind reviewed plan hashes and cost allocations into shared reservations, and verify the active hold under the ledger lock before claiming or launching a Job.
Hash the frozen task tree rather than a restored live source. Require verified file inventories for remote identities and carry acquisition hashes through the discovery pipeline.
Reuse native file-count and byte caps before downloading selected tasks and shared assets. Verify pinned file inventories, stream Harbor downloads under a byte cap, and check bounded tar contents before materializing task files.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 6 total unresolved issues (including 4 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Deadline discards completed job status
- Removed the post-inspection deadline override so a verified completed controller remains completed.
- ✅ Fixed: Submission receipts skip durable fsync
- Receipt writes now flush and fsync the temporary file before replacement and fsync the parent directory afterward.
Or push these changes by commenting:
@cursor push 771b963ef7
Preview (771b963ef7)
diff --git a/examples/hub_validation_pipeline/cloud.py b/examples/hub_validation_pipeline/cloud.py
--- a/examples/hub_validation_pipeline/cloud.py
+++ b/examples/hub_validation_pipeline/cloud.py
@@ -769,8 +769,16 @@
def _write_receipt(path, value):
temporary = path.with_suffix(".tmp")
- temporary.write_bytes(_json_bytes(value))
- temporary.replace(path)
+ with temporary.open("wb") as stream:
+ stream.write(_json_bytes(value))
+ stream.flush()
+ os.fsync(stream.fileno())
+ os.replace(temporary, path)
+ directory = os.open(path.parent, os.O_RDONLY)
+ try:
+ os.fsync(directory)
+ finally:
+ os.close(directory)
def stage_inputs(plan_path, *, api=None, secrets=None, expected_plan_sha256=None):
diff --git a/examples/hub_validation_pipeline/prepared_batch.py b/examples/hub_validation_pipeline/prepared_batch.py
--- a/examples/hub_validation_pipeline/prepared_batch.py
+++ b/examples/hub_validation_pipeline/prepared_batch.py
@@ -54,8 +54,6 @@
or job.owner.name != owner
):
raise ValueError("Controller identity mismatch")
- if clock() >= deadline:
- return "TIMEOUT"
return job.status.stage if job.status.stage in STAGES else "UNKNOWN"You can send follow-ups to the cloud agent here.
…validation-examples
…validation-examples
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Batch launcher rejects prepared manifests
- Preparation writers now persist relative plan_path values and bound plan_sha256 digests that prepared_batch can validate and launch.
- ✅ Fixed: Worker omits required Hub pin
- Remote bootstrap now installs huggingface_hub==1.33.0 into the synced worker environment before starting the worker.
Or push these changes by commenting:
@cursor push 0aff254920
You can send follow-ups to the cloud agent here.
…validation-examples
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Non-positive poll deadline disables timeout
- Clamped alarm durations to one microsecond and protected timer setup with cleanup so expired poll budgets cannot disable or leak the controller timeout.
Or push these changes by commenting:
@cursor push 3a695019ec
Preview (3a695019ec)
diff --git a/examples/coding_audit/hf.py b/examples/coding_audit/hf.py
--- a/examples/coding_audit/hf.py
+++ b/examples/coding_audit/hf.py
@@ -72,6 +72,7 @@
# the main Unix thread so a blocked synchronous SDK call can be interrupted.
if threading.current_thread() is not threading.main_thread():
raise HFBackendError("HF coding controller requires the main thread")
+ seconds = max(0.000001, seconds)
previous_handler = signal.getsignal(signal.SIGALRM)
previous_timer = signal.getitimer(signal.ITIMER_REAL)
started = time.monotonic()
@@ -79,13 +80,13 @@
def expired(signum, frame):
raise _DeadlineExceeded("controller deadline exceeded")
- signal.signal(signal.SIGALRM, expired)
- signal.setitimer(
- signal.ITIMER_REAL,
- min(seconds, previous_timer[0]) if previous_timer[0] else seconds,
- 0.1,
- )
try:
+ signal.signal(signal.SIGALRM, expired)
+ signal.setitimer(
+ signal.ITIMER_REAL,
+ min(seconds, previous_timer[0]) if previous_timer[0] else seconds,
+ 0.1,
+ )
yield
finally:
signal.setitimer(signal.ITIMER_REAL, 0)You can send follow-ups to the cloud agent here.
…validation-examples
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 5 total unresolved issues (including 3 from previous reviews).
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Acquisition hashes disagree with bundle tree
- Acquisition verification now applies the bundle's excluded-directory policy before comparing captured task hashes.
- ✅ Fixed: Usage dropped before receipt fallback
- Target responses now pass typed usage directly to the receipt fallback so billed token counts survive non-JSON provider details.
Or push these changes by commenting:
@cursor push 4cb0068c7e
Preview (4cb0068c7e)
diff --git a/examples/coding_audit/hf.py b/examples/coding_audit/hf.py
--- a/examples/coding_audit/hf.py
+++ b/examples/coding_audit/hf.py
@@ -753,7 +753,7 @@
parsed = client.chat_completion(**request)
response = _plain(parsed)
# Access the typed nested value: outer SDK dicts can omit extras.
- response["usage"] = _plain(parsed.usage)
+ response["usage"] = parsed.usage
choices = response.get("choices")
receipt = {
"model": response.get("model"),
diff --git a/examples/coding_audit/test_target_budget.py b/examples/coding_audit/test_target_budget.py
--- a/examples/coding_audit/test_target_budget.py
+++ b/examples/coding_audit/test_target_budget.py
@@ -119,6 +119,22 @@
assert receipt["response_validated"] and not receipt["discarded"]
+def test_non_json_target_usage_retains_billed_counts(harness):
+ h = harness
+ parsed = response("saved")
+ parsed.usage["provider_detail"] = object()
+ h.inference.responses = [parsed]
+ solved = h.backend.solve(model(), 0)
+ assert solved["error"] == "provider returned invalid usage details"
+ assert solved["usage"] == {"input_tokens": 10, "output_tokens": 5}
+ receipt = solved["model_metadata"]["responses"][0]
+ assert receipt["usage_counts"] == {
+ "prompt_tokens": 10,
+ "completion_tokens": 5,
+ "total_tokens": 15,
+ }
+
+
@pytest.mark.parametrize(
"problem", ["missing_total", "bad_total", "nonfinite_detail", "oversized_detail"]
)
diff --git a/examples/hub_validation_pipeline/cloud.py b/examples/hub_validation_pipeline/cloud.py
--- a/examples/hub_validation_pipeline/cloud.py
+++ b/examples/hub_validation_pipeline/cloud.py
@@ -256,19 +256,23 @@
raise ValueError("Remote source requires verified acquisition file hashes")
return
expected = source["task_file_hashes"]
- if (
- not isinstance(expected, dict)
- or set(expected) != set(task_files)
- or any(
- not isinstance(value, str) or not re.fullmatch(r"[0-9a-f]{64}", value)
- for value in expected.values()
- )
- or any(
- hashlib.sha256(data).hexdigest() != expected[name]
- for name, data in task_files.items()
- )
+ if not isinstance(expected, dict) or any(
+ not isinstance(name, str)
+ or not isinstance(value, str)
+ or not re.fullmatch(r"[0-9a-f]{64}", value)
+ for name, value in expected.items()
):
raise ValueError("Captured task files disagree with verified acquisition")
+ captured = {
+ name: value
+ for name, value in expected.items()
+ if not any(part in _EXCLUDED for part in PurePosixPath(name).parts)
+ }
+ if set(captured) != set(task_files) or any(
+ hashlib.sha256(data).hexdigest() != captured[name]
+ for name, data in task_files.items()
+ ):
+ raise ValueError("Captured task files disagree with verified acquisition")
def _config(config):
diff --git a/examples/hub_validation_pipeline/test_cloud.py b/examples/hub_validation_pipeline/test_cloud.py
--- a/examples/hub_validation_pipeline/test_cloud.py
+++ b/examples/hub_validation_pipeline/test_cloud.py
@@ -669,12 +669,25 @@
def test_task_exclusions_match_runner_source_identity(inputs, tmp_path):
from openenv.validation.runner import source_digest
- task = inputs[1]
+ repo, task, config = inputs
for name in (".git", ".worktrees", ".venv", "__pycache__", "outputs"):
(task / name).mkdir()
(task / name / "binary-secret").write_bytes(b"\x00ignored")
before = source_digest(task)
- output, plan = prepared(inputs, tmp_path)
+ source = {
+ "kind": "hf",
+ "revision": SHA,
+ "task_path": ".",
+ "task_file_hashes": {
+ path.relative_to(task).as_posix(): hashlib.sha256(
+ path.read_bytes()
+ ).hexdigest()
+ for path in task.rglob("*")
+ if path.is_file()
+ },
+ }
+ output = tmp_path / "prepared"
+ plan = cloud.prepare(task, config, output, repo_root=repo, source=source)
destination = tmp_path / "snapshot"
cloud.extract_bundle(
(output / "bundle.zip").read_bytes(), destination, plan["bundle_sha256"]You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit fe9363b. Configure here.
|
Independent read-only final-head review gate: both reviewers APPROVE |
|
Independent final-head delta review APPROVE for |
…validation-examples
…validation-examples




Summary
Adds the reproducible Hub pipeline, a tiny local control demo, explicit operator-supplied evaluation panels, immutable cloud bundles and cohort launch scripts. Native admission verifies standard semantic results and raw reward/replay/canary artifacts, scales tolerance and both floor gaps by the reward-range width, binds the startup-recorded retained image and accepts reviewed Dockerfile builds or digest-pinned prebuilt images without rewriting acquired tasks. Mutable image tags remain excluded. Missing oracles remain semantic failures for unsupported startup shapes.
Cloud plans bind acquired file hashes, consumed plan hashes, runtime versions and exact shared-spend allocations. Both preparation producers emit the relative plan path and exact hash consumed by the prepared-cohort launcher. The remote bootstrap performs frozen sync, installs and verifies its declared HF SDK pin before starting the worker. Acquired native assets under execution-excluded paths are explicitly unsupported rather than silently filtered. Expired deadlines prevent SDK dispatch; malformed provider metadata retains available integer billing counts before response normalization. Uncertain launches keep durable receipts and reservations for reconciliation.
Based on main after #1591 (which includes the operator coding audit from #1592) and #1607 (digest-pinned prebuilt/image-only Harbor tasks). The Git-pinned Harbor integration depends on #1304 (JianYan11's FastMCP 4 migration), which remains an external dependency rather than silently incorporating its commits.
HF comparison: discovery uses the conjunction
library:harborandrl-environment; acquisition resolves one full revision and uses normal SDK revision/path semantics. Uploads add files through ordinary Hub commits with optimistic parent checks, matchinghf upload's additive behavior. They do not mirror-delete remote files. Jobs receive runtime secrets, immutable inputs and explicit flavor/timeout settings.Evidence
d65475d3b6c978c19cf62e0c6ce0b731fe16cbf2. Public headbaab8461c806c23d3832b978cc5d166e49dafb87; fast-forward push only. The previous exact-head merge authorization was not used after main advanced; this new head needs the user's merge decision.1c1405859, 2,895 broader CPU tests passed, 8 skipped and 58 deselected across validation, both examples and the trial hook. Live/network/Docker marks were excluded.uv lockanduv lock --checkpass; no manual lockfile resolution.1c1405859: correctness (448 scoped tests) and spec/public boundary (442 scoped tests). A fresh independent final-head review APPROVE covers the six-file1c1405859..9c2c0b3dbquality-review delta (249 scoped tests). Neither review asserts live-provider, Docker, billing or training qualification.Merge Danger
Door: two-way
Blast Radius: examples
Explicit submission incurs cloud/provider costs. Reservations model configured workloads rather than enforcing an account billing cap. Catalog enumeration is metadata, not runtime qualification. Required checks and any FAIL/ERROR block admission; supplemental audit success does not establish full RFC compliance, hostile-code verifier integrity or learning benefit. Runtime-pinning changes require fresh preparation/review of older plans. This PR remains open for the user's merge decision.
Main-refresh compatibility: new source_image metadata is additive; the schema-2 trial overlays, retained-image handoff and inspected source/config/image/cleanup bindings are unchanged. Six independent offline compatibility cases passed. Example README and pipeline plan now describe pinned prebuilt/image-only support accurately; no admission/runtime relaxation was needed.