Skip to content

Docker runner: provision the Forge engine; harness guard: refuse added runtime patching and library operator calls - #134

Merged
irvineoy merged 4 commits into
AMD-AGI:mainfrom
zili-amd:forge-runtime-and-harness-guards
Oct 27, 2026
Merged

irvineoy merged 4 commits into
AMD-AGI:mainfrom
zili-amd:forge-runtime-and-harness-guards

Conversation

@zili-amd

@zili-amd zili-amd commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Provision the Forge engine in Docker and reject added runtime patching or library delegation before candidate scoring. Concurrent Forge campaigns keep their own engine installation, and custom C++ methods remain usable.

  • Resolve the committed engine revision before preflight and worker startup. Install its exported source into a staging directory, lock setup for that revision, and publish atomically. Python cache tag, container image identity and engine revision select separate cache directories; setup never removes an active campaign's installation.
  • Support both the keyword implementer API and the grouped ImplementerSetup API. Preserve task instructions, protected paths, correctness-only gates, usage and time limits. After initialization, refresh the loop's remaining budget and forward the same absolute deadline when supported.
  • Normalize bare, qualified and imported-alias reflective builtins. Mask C++ comments and literals correctly, including URL strings, raw strings and line continuations. Identify Tensor member operators without rejecting same-named methods on custom types.
  • Record the effective source policy in workspace context. Old snapshots without initial site counts explicitly report this policy as disabled.

The shared guard applies to every agent's evaluation. Non-Forge runs receive no new engine mount or engine environment. The runner also forwards ANTHROPIC_CUSTOM_HEADERS for GEAK/Forge and provides a noninteractive Docker exec command. No task packages, numerical tolerances, workloads or scoring/timing helpers are changed.

Validation

  • Linux focused guard, lifecycle, GEAK and Forge regression suites: 935 passed, 32 skipped. The skips cover optional installed-engine checks and existing gated GEAK cases.
  • Real installed engine bb2da63332dcbc46bdf10702c516ef43eb80a016 (1.1.4), Linux amd64 / Python 3.12: 109 passed in the Forge integration suites, including initialization retries, harness tampering, timeout, complete candidate delivery and loop handoff. Provider responses are scripted CPU fixtures; these are not model-driven GPU optimization results. The new API/budget mapping cases also passed separately.
  • Docker runner mock suite: passed. Python compilation and git diff --check: passed.
  • Fresh Fleet GPU tests on MI350X, gfx950, in the pinned ROCm development image: native HIP GELU and the shared TaskSession baseline/candidate compile, correctness and performance actions passed. The custom Accumulator::sum implementation remained allowed and numerically correct. An added ATen call after a URL string was rejected, and a zero-output GPU mutant failed correctness. Jobs job-48b54231 and job-5d97db84 exited 0; task-owned resources are released.
  • GPU replay checks used a 1 GiB read-only eviction sweep before each complete graph replay, with replay outputs checked outside timing. Raw samples and source/runtime receipts are retained. This is a small integration fixture, not a speedup claim or retained-task qualification.

Limits

The source check is structural, not a C++ type checker or security sandbox. Initial per-file site counts remain the comparison boundary; semantic task review and numerical checks are still required. The grouped engine's new-file allowlist supports helpers in existing directories of declared editable trees; helpers in newly created directories are rejected by that engine.

A fresh model-driven Forge optimization campaign and other agents' GPU score comparisons were not rerun. The earlier author's MI300X campaign evidence applies to 11cc060d; it is separate from the fresh tests above. Full repository CI must finish on the current head before merging.

zili-amd and others added 2 commits October 21, 2026 11:38
…n it

A forge run config now gets what a GEAK run gets: its agent backend's CLI
(Claude unless the run sets agent_backend: codex), the pinned Hyperloom
checkout named by HYPERLOOM_ROOT mounted read-only, and Claude credentials
forwarded by name. `make docker-setup-forge` installs Hyperloom[forge] from
an exported copy of that checkout's committed HEAD into the host-mounted user
base, records the installed revision, and skips an unchanged one; the
container bootstrap prepends that engine only for Forge runs. Other agents
never receive the checkout, and setup-forge receives no credentials.

`docker_benchmark.sh exec <command>` runs one command non-interactively in the
runtime container, and ANTHROPIC_CUSTOM_HEADERS is forwarded with the other
Anthropic variables to GEAK and Forge containers.

Co-authored-by: Cursor <cursoragent@cursor.com>
… operator calls

Every evaluation path verifies the workspace harness before it trusts a
measurement. That check now also refuses a candidate file that adds, relative
to the task's starting file:

- runtime patching in Python (exec/eval/setattr/globals, frame and code
  object access, importlib reloads), which can make a protected launcher,
  reference or timing loop run differently from how it reads;
- library operator calls in C++/HIP sources (hipBLAS, rocBLAS, Tensile,
  MIOpen, and the ATen/torch products, softmax, norms and reductions a task
  asks the candidate to implement);
- PyTorch or library operators in the host code of a Triton candidate
  (bodies of @triton.jit functions are skipped).

Sites the task's own starting file already has stay allowed. The per-file
starting counts are recorded with the task session, so a resumed invocation
compares against the same state, and GEAK's bridge carries them through its
saved snapshot.

Co-authored-by: Cursor <cursoragent@cursor.com>

Two existing tests change with it. An actual-gate payload such as
`torch.allclose = lambda *args, **kwargs: True` was accepted under a
whole-file editable scope and refused only under the symbol scope; it is now
refused under both. In the softmax builder test, `vars(sys)` and
`sys.__dict__` reads are refused by the guard before the task's own
dependency check, which still rejects every listed statement.

Co-authored-by: Cursor <cursoragent@cursor.com>
@zili-amd

Copy link
Copy Markdown
Contributor Author

@irvineoy This is ready for review. Since opening it I ran a smoke on this exact commit on MI300X: make docker-setup-forge, docker_benchmark.sh exec, and a 45-minute Forge run end to end. The results are under Validation.

Pin the engine revision before Docker workers start, and publish installations
atomically into separate Python/image/revision directories. Preserve active
campaigns when another setup uses a different engine or image.

Recognize qualified reflective builtins, mask C++ literals correctly, and limit
member-operator checks to identifiable Tensor receivers. Document the structural
check and expose its effective policy in saved workspace context.

Validation: 262 Linux CPU regression tests passed, 9 optional engine tests skipped;
110 focused tests passed on macOS; Docker runner mock suite passed. Fresh Fleet
GPU integration remains pending runtime/transfer availability.
Map initialization settings to the engine's task, measurement and session
objects while retaining the keyword API. Preserve protected paths, correctness
checks, timeout limits and declared new-file boundaries. Refresh the loop budget
after initialization and forward its absolute deadline when supported.

Update installed-engine regression fixtures for the public evaluation and
campaign interfaces. Publish the process-cleanup fixture's PID atomically.

Validation: 109 installed-engine tests passed with a scripted provider on Linux
amd64, plus six API and budget mapping unit cases. Fleet MI350X native HIP and
shared TaskSession compile/correctness/performance checks passed, including
illegal library-call and wrong-output controls. No task packages were changed.
@irvineoy
irvineoy merged commit 8ef210a into AMD-AGI:main Oct 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants