pr_diff bakes the oracle into the agent's image, so the agent can read and apply it for a perfect reward

[surajsharan]

Description

Every pr_diff task's environment/Dockerfile base64-bakes the oracle patch, the instruction, and the verifier into /verifier/ in the image (pr_diff.py, build_pr_diff_environment_dockerfile):

RUN echo "<b64>" | base64 -d > /verifier/oracle.patch
RUN echo "<b64>" | base64 -d > /verifier/instruction.md
RUN echo "<b64>" | base64 -d > /verifier/verifier.py

That is the same container the agent gets a shell in, and it runs as root. So the agent can read /verifier/oracle.patch — the reference fix — and apply it. tests/test.sh then scores the agent's diff against that same oracle, yielding reward = 1.0 on every deterministic component without the agent solving anything. For an RL training environment this is a reward-hack a policy under optimization pressure will find.

The runtime pipelines already avoid this: pr_runtime ships its grading assets as plain tests/ files (_runtime_aux_files), and Harbor uploads a task's tests/ into the container only at verification time, never during the agent phase. pr_diff predates that pattern — the baking landed in #40, the plain-tests/ pattern in #45 — and was never migrated. The same class of leak was fixed for code_instruct / equivalence_tests in #55.

Reproduction

The Dockerfile builder bakes the oracle directly, so no network or dataset is needed:

from repo2rlenv.pipelines.pr_diff import build_pr_diff_environment_dockerfile
df = build_pr_diff_environment_dockerfile(
    repo_url="https://github.com/pallets/click",
    base_commit="<sha>",
    oracle_diff="diff --git a/x b/x\n+the_fix\n",
    instruction="...",
)
assert "base64 -d > /verifier/oracle.patch" in df   # the oracle is in the agent's image

Building any emitted task's environment/ and shelling into it shows the same — the agent's container has /verifier/oracle.patch, and git apply /verifier/oracle.patch followed by the baked tests/test.sh scores reward.txt = 1.000000, all six components 1.0. (This also holds for any published pr_diff task, e.g. AdithyaSK/repo2rlenv-pr-diff.)

Expected behavior

The oracle must not be reachable from the agent's environment. Ship the oracle, verifier, and instruction as tests/ aux files (the pr_runtime pattern) so Harbor delivers them only at verify time; the agent's image contains the repo at base_commit and nothing else.

Impact

Every pr_diff task generated to date is affected, including the published 181-task reference dataset. Existing datasets keep the verifier they were emitted with, so they need re-emission to pick up the fix; the generation-side change makes future tasks safe.

Residual (separate)

Even with the oracle out of the image, the verifier still runs in the agent's own container (shared mode), so PYTHONPATH / sitecustomize / shadowed-binary tampering is a distinct, deeper vector. Harbor's [verifier] environment_mode = "separate" is the fix for that and is a larger change; I'd keep it to a follow-up.

I have a fix with tests, verified end to end through Harbor; PR to follow.