Repository navigation
Docker runner: provision the Forge engine; harness guard: refuse added runtime patching and library operator calls - #134
Merged
irvineoy merged 4 commits intoOct 27, 2026
Conversation
…n it A forge run config now gets what a GEAK run gets: its agent backend's CLI (Claude unless the run sets agent_backend: codex), the pinned Hyperloom checkout named by HYPERLOOM_ROOT mounted read-only, and Claude credentials forwarded by name. `make docker-setup-forge` installs Hyperloom[forge] from an exported copy of that checkout's committed HEAD into the host-mounted user base, records the installed revision, and skips an unchanged one; the container bootstrap prepends that engine only for Forge runs. Other agents never receive the checkout, and setup-forge receives no credentials. `docker_benchmark.sh exec <command>` runs one command non-interactively in the runtime container, and ANTHROPIC_CUSTOM_HEADERS is forwarded with the other Anthropic variables to GEAK and Forge containers. Co-authored-by: Cursor <cursoragent@cursor.com>
… operator calls Every evaluation path verifies the workspace harness before it trusts a measurement. That check now also refuses a candidate file that adds, relative to the task's starting file: - runtime patching in Python (exec/eval/setattr/globals, frame and code object access, importlib reloads), which can make a protected launcher, reference or timing loop run differently from how it reads; - library operator calls in C++/HIP sources (hipBLAS, rocBLAS, Tensile, MIOpen, and the ATen/torch products, softmax, norms and reductions a task asks the candidate to implement); - PyTorch or library operators in the host code of a Triton candidate (bodies of @triton.jit functions are skipped). Sites the task's own starting file already has stay allowed. The per-file starting counts are recorded with the task session, so a resumed invocation compares against the same state, and GEAK's bridge carries them through its saved snapshot. Co-authored-by: Cursor <cursoragent@cursor.com> Two existing tests change with it. An actual-gate payload such as `torch.allclose = lambda *args, **kwargs: True` was accepted under a whole-file editable scope and refused only under the symbol scope; it is now refused under both. In the softmax builder test, `vars(sys)` and `sys.__dict__` reads are refused by the guard before the task's own dependency check, which still rejects every listed statement. Co-authored-by: Cursor <cursoragent@cursor.com>
zili-amd
force-pushed
the
forge-runtime-and-harness-guards
branch
from
October 21, 2026 19:06
cca3421 to
11cc060
Compare
This was referenced Oct 21, 2026
zili-amd
marked this pull request as ready for review
October 26, 2026 20:44
zili-amd
requested review from
irvineoy,
sharareh-y and
sinarafati-amd
as code owners
October 26, 2026 20:44
Contributor
Author
|
@irvineoy This is ready for review. Since opening it I ran a smoke on this exact commit on MI300X: |
Pin the engine revision before Docker workers start, and publish installations atomically into separate Python/image/revision directories. Preserve active campaigns when another setup uses a different engine or image. Recognize qualified reflective builtins, mask C++ literals correctly, and limit member-operator checks to identifiable Tensor receivers. Document the structural check and expose its effective policy in saved workspace context. Validation: 262 Linux CPU regression tests passed, 9 optional engine tests skipped; 110 focused tests passed on macOS; Docker runner mock suite passed. Fresh Fleet GPU integration remains pending runtime/transfer availability.
Map initialization settings to the engine's task, measurement and session objects while retaining the keyword API. Preserve protected paths, correctness checks, timeout limits and declared new-file boundaries. Refresh the loop budget after initialization and forward its absolute deadline when supported. Update installed-engine regression fixtures for the public evaluation and campaign interfaces. Publish the process-cleanup fixture's PID atomically. Validation: 109 installed-engine tests passed with a scripted provider on Linux amd64, plus six API and budget mapping unit cases. Fleet MI350X native HIP and shared TaskSession compile/correctness/performance checks passed, including illegal library-call and wrong-output controls. No task packages were changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Provision the Forge engine in Docker and reject added runtime patching or library delegation before candidate scoring. Concurrent Forge campaigns keep their own engine installation, and custom C++ methods remain usable.
ImplementerSetupAPI. Preserve task instructions, protected paths, correctness-only gates, usage and time limits. After initialization, refresh the loop's remaining budget and forward the same absolute deadline when supported.The shared guard applies to every agent's evaluation. Non-Forge runs receive no new engine mount or engine environment. The runner also forwards
ANTHROPIC_CUSTOM_HEADERSfor GEAK/Forge and provides a noninteractive Dockerexeccommand. No task packages, numerical tolerances, workloads or scoring/timing helpers are changed.Validation
bb2da63332dcbc46bdf10702c516ef43eb80a016(1.1.4), Linux amd64 / Python 3.12: 109 passed in the Forge integration suites, including initialization retries, harness tampering, timeout, complete candidate delivery and loop handoff. Provider responses are scripted CPU fixtures; these are not model-driven GPU optimization results. The new API/budget mapping cases also passed separately.git diff --check: passed.TaskSessionbaseline/candidate compile, correctness and performance actions passed. The customAccumulator::sumimplementation remained allowed and numerically correct. An added ATen call after a URL string was rejected, and a zero-output GPU mutant failed correctness. Jobsjob-48b54231andjob-5d97db84exited 0; task-owned resources are released.Limits
The source check is structural, not a C++ type checker or security sandbox. Initial per-file site counts remain the comparison boundary; semantic task review and numerical checks are still required. The grouped engine's new-file allowlist supports helpers in existing directories of declared editable trees; helpers in newly created directories are rejected by that engine.
A fresh model-driven Forge optimization campaign and other agents' GPU score comparisons were not rerun. The earlier author's MI300X campaign evidence applies to
11cc060d; it is separate from the fresh tests above. Full repository CI must finish on the current head before merging.