Skip to content

examples: add bounded coding audits with HF Sandboxes - #1592

Merged
burtenshaw merged 5 commits into
ben/rfc008-validation-completionfrom
ben/coding-audit-operator
Nov 10, 2026
Merged

burtenshaw merged 5 commits into
ben/rfc008-validation-completionfrom
ben/coding-audit-operator

Conversation

@burtenshaw

@burtenshaw burtenshaw commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

reviewed task + operator configuration
  → provided/generated references + fresh verifier replays
  → trusted wrong-answer controls + bounded leakage probes
  → repeated target attempts + difficulty statistics
  → checksummed evidence and an explicit acceptance decision

Adds examples/coding_audit and its standalone CLI. The scorer checks exact reward thresholds and task/verifier/image/execution bindings. The HF backend keeps inference in the controller, transfers only declared submission files to fresh graders, and verifies terminal cleanup. Candidates can opt into declared, bounded public starter files with recorded hashes. Generated references remain candidates; target difficulty is measured separately from verifier reliability.

This is operator-side measurement under RFC 008, based on #1591. It adds no installed validation API. The complete Hub workflow's Git-pinned Harbor installation also depends on #1304, JianYan11's FastMCP 4 migration; that migration is an external dependency.

HF comparison: execution uses HfApi, Sandbox and InferenceClient, with explicit image, flavor and timeout settings corresponding to hf jobs conventions. Provider selection is explicit; model source revisions do not attest provider-served weights. The tested HF SDK is 1.33.0.

Evidence

  • Before: no example collected and bound this reference/control/replay/target evidence.
    After: 407 offline scoring, collector, actual-SDK serialization, stream completion, context binding, deadline, redaction and cleanup tests pass. Commands are in the example README.
  • Whole-repository hooks encounter existing formatting differences and local optional-tool/dependency setup gaps. Changed-file lint/import/format checks pass; focused test results are recorded separately.

Merge Danger

Door: two-way

Blast Radius: examples

Invoking the live backend launches paid services. Use reviewed tasks: Sandbox RPC exposes its execution bearer to workload-controlled code, and separate verifier transfer does not establish hostile-code isolation. The report explicitly leaves full RFC qualification and trainability unestablished.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread examples/coding_audit/scoring.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Nested sandbox mkdir may fail
    • Sandbox workdirs and reward directories are now created ancestor-first, with strict-parent regression coverage.

Create PR

Or push these changes by commenting:

@cursor push ec069c8970
Preview (ec069c8970)
diff --git a/examples/coding_audit/hf.py b/examples/coding_audit/hf.py
--- a/examples/coding_audit/hf.py
+++ b/examples/coding_audit/hf.py
@@ -108,6 +108,14 @@
     return path == parent or path.startswith(parent + "/")
 
 
+def _mkdir(files, path):
+    path = PurePosixPath(path)
+    for directory in reversed((path, *path.parents)):
+        directory = str(directory)
+        if directory != "/" and not files.exists(directory):
+            files.mkdir(directory)
+
+
 def _candidate_script(raw):
     lines = raw.lstrip().splitlines(keepends=True)
     opening = lines[0].strip()
@@ -395,7 +403,7 @@
             raise HFBackendError("sandbox control-plane readiness did not converge")
         # The SDK stat/read pair otherwise permits up to 512 MiB per read.
         sandbox.files.MAX_READ_BYTES = self.max_bytes
-        sandbox.files.mkdir(self.workdir)
+        _mkdir(sandbox.files, self.workdir)
         return sandbox
 
     def _recover_startup_jobs(self, record, pending):
@@ -1191,7 +1199,7 @@
                     sandbox.files.write(path, raw)
                 # Harbor normally mounts this directory. Fresh Sandbox graders
                 # need it before a native verifier can write its reward file.
-                sandbox.files.mkdir(str(PurePosixPath(self.reward_path).parent))
+                _mkdir(sandbox.files, PurePosixPath(self.reward_path).parent)
                 # A stale image-provided reward must not become new evidence.
                 if sandbox.files.exists(self.reward_path):
                     sandbox.files.delete(self.reward_path)

diff --git a/examples/coding_audit/test_hf.py b/examples/coding_audit/test_hf.py
--- a/examples/coding_audit/test_hf.py
+++ b/examples/coding_audit/test_hf.py
@@ -171,6 +171,8 @@
         self.writes.append((path, data, mode))
 
     def mkdir(self, path):
+        if str(PurePosixPath(path).parent) not in self.directories:
+            raise SandboxError("missing parent", status_code=404)
         self.directories.add(path)
 
     def delete(self, path):

You can send follow-ups to the cloud agent here.

Reviewed by Cursor Bugbot for commit c1fc62a. Configure here.

Comment thread examples/coding_audit/hf.py Resolved
Read original task files through the descriptor-based reader before writing the frozen copy, so copying cannot erase hard-link evidence. Cover context, instruction, verifier and reference artifacts through the runner with offline SDK fakes.
Require a nonempty tool-call list when the provider finishes with tool_calls. Exercise null, missing and empty calls through collection and scoring after a valid terminal action, and verify failed targets are never graded.
Include the repository root in PYTHONPATH so pytest can import the examples package. Verified the documented command with all 417 offline tests passing.
burtenshaw added a commit that referenced this pull request Nov 10, 2026
Takes the coding-audit fixes for linked artifacts, empty tool-call completions and the offline test path (c1fc62a..4278afa); no file overlaps this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@burtenshaw
burtenshaw merged commit 50a4a27 into ben/rfc008-validation-completion Nov 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature size: extra-large Extra-large pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant