pr_diff: ship oracle + verifier via tests/, not baked in image - #145
Open
surajsharan wants to merge 1 commit into
Open
surajsharan wants to merge 1 commit into
surajsharan wants to merge 1 commit into
Conversation
The environment/Dockerfile base64-baked the oracle patch, instruction,
and verifier into /verifier/ in the agent's own image. Since the agent
has a root shell there, it could read /verifier/oracle.patch and git
apply it for a perfect reward without solving anything.
Ship them as tests/{oracle.patch,verifier.py,instruction.md} aux files
instead. Harbor uploads a task's tests/ only at verify time, so the
oracle never enters the agent's image. This is the plain-tests pattern
pr_runtime has used since huggingface#45; pr_diff predated it (baked in huggingface#40).
Also clear /logs/verifier/reward.txt before the verifier runs, so an
agent that breaks python3 and pre-writes it can't pin the reward — the
verifier is the sole authority. And teach validate --deep to require the
tests/ verifier assets on a runnable pr_diff task.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pr_diffbase64-baked the oracle patch, instruction, and verifier into/verifier/in the agent's own image, so an agent couldgit apply /verifier/oracle.patchand score a perfect reward without solving anything. They now ship astests/{oracle.patch, verifier.py, instruction.md}aux files, which Harbor uploads only at verify time — the exact plain-tests/patternpr_runtimehas used since pr_runtime: graded F2P/P2P reward + 100-env dataset (Arc 2) #45, and the same class of fix as fix(code_instruct/equivalence_tests): grading test unreachable to non-oracle agents #55. The agent's image now contains the repo atbase_commitand nothing else.pr_diff'stest.shhad no guard, so an agent that brokepython3and pre-wrote/logs/verifier/reward.txtkept its value.test.shnowrm -fs the reward files first, mirroring Harbor's own separate-verifierempty_dirs([verifier_dir]). The verifier is the sole authority on the score.validate --deepnow checks a runnablepr_difftask shipstests/{verifier.py, oracle.patch, instruction.md}(parity with thepr_runtimegraded-verifier check), so a task that would build but score nothing is caught statically.pipelines/README.mdandpr_diff.md, and RFC 0001, described the baked/verifier/layout that this removes.Test plan
uv run --all-extras pytest -q— 1933 passed, 10 skipped (new/updated tests intest_pipeline_pr_diff.py,test_validation.py,test_pr_diff_clone.py)ruff check .+ruff format --check .clean;mkdocs buildclean (0 warnings)repo2rlenv validate --deeppasses on a freshly emitted task and flags one with thetests/assets removedmain, absent after (Docker Desktop / WSL2). On a task emitted with this branch's builders, the built image has no/verifier, no/tests, and nooracle.patchanywhere —find / -name oracle.patchis empty — so thegit apply /verifier/oracle.patchexploit is impossible.harbor run -a oracle -e dockeron the fixed task →reward.txt = 1.000000, all deterministic components 1.0,judge_status: no_api_key(renormalized). The verifier runs correctly from thetests/upload delivered at verify time.Out of scope
pr_difftasks (includingAdithyaSK/repo2rlenv-pr-diff) keep their baked verifier and need re-emission to pick up the fix. A migration/re-bake tool could be a follow-up — and, since the verifier is now a plaintests/verifier.pyfile, upgrading it in place becomes a file swap rather than an image rebuild.PYTHONPATH/sitecustomize/ shadowed-binary tampering remains. The fix is Harbor's[verifier] environment_mode = "separate", a larger change worth its own discussion.Closes #144