pr_diff verifier: judge via any OpenAI-compatible endpoint

[surajsharan]

Summary

• The pr_diff verifier's llm_judge was hardwired to Anthropic's Messages API and ANTHROPIC_API_KEY, so a self-hosted model could never act as judge and 0.50 of the reward weight was unavailable without an Anthropic key. Two new verifier env vars, settable per run via harbor run --ve, fix that: R2E_JUDGE_ENDPOINT (base URL of a vLLM / Ollama / llama.cpp / gateway server; the request goes to <endpoint>/chat/completions with a bearer token) and R2E_JUDGE_API_KEY (optional). R2E_JUDGE_MODEL is required alongside an endpoint since a self-hosted server has no Haiku.
• Same key rule as cli: add --llm-endpoint/--llm-key-env, keyless self-hosted LLMs #104: with an endpoint set, ANTHROPIC_API_KEY is never forwarded to the custom server; a placeholder bearer is sent unless R2E_JUDGE_API_KEY names one. The default Anthropic route is unchanged byte-for-byte, and the verifier stays stdlib-only (it is baked into a bare python:3.12-slim image) — a new test parses the baked source and asserts every import is stdlib.
• The endpoint route sends temperature: 0. At the server's default temperature Qwen3.5-4B scored the same oracle-as-prediction input 0.5 / 1.0 / unparseable across three calls; at 0 it is deterministic and correct (below). The Anthropic route keeps its defaults — the published weights were calibrated against them.
• reward-details.json gains judge_endpoint (null on the Anthropic route) so runs are auditable; judge_status gains no_judge_model for the endpoint-without-model case.
• Docs corrected to match the code: R2E_JUDGE_MODEL is a bare model id as the serving API names it, not a LiteLLM-qualified name (ENV.md); judge_model example and the judge_status value list in REWARD_SCHEMA.md; RFC 0001's "LLM judge via LiteLLM"; the "always uses Anthropic" comments in the pipeline docs and the dataset-card text in hub.py. pr_diff.md gets a self-hosted-judge harbor run example.

Test plan

• uv run --all-extras pytest -q — 1880 passed, 7 skipped (py3.12, all extras as CI installs them) (13 new tests in tests/test_pr_diff_verifier.py for the two request routes, response parsing, and the env → config mapping; 1 new stdlib-only guard in tests/test_pipeline_pr_diff.py)
• uv run ruff check . and uv run ruff format --check . clean
• mkdocs build clean (zero warnings)
• Reproduced on main first: llm_judge(model="Qwen/Qwen3.5-4B", api_key="EMPTY") builds a request to https://api.anthropic.com/v1/messages with an x-api-key header, has no endpoint parameter, and a real call returns (None, "network")
• Live against vLLM 0.20.2 serving Qwen/Qwen3.5-4B on an RTX 3090, verifier loaded standalone, a real pallets/click commit as oracle + its message as the instruction:
  • predicted = oracle → score 1.0, ok (3/3 runs, ~1 s each)
  • predicted = an unrelated commit's diff → score 0.0, ok (3/3 runs)
  • endpoint unreachable → network
  • same through _judge_config_from_env() with ANTHROPIC_API_KEY also set → bearer sent EMPTY, score 1.0, ok; vLLM's access log shows the calls
  • the untruncated 4.3k-char oracle and the 12.8k-char one (truncated to 4k by the verifier) both judged correctly at temperature 0
• End to end through Harbor (Docker Desktop on WSL2, harbor run -a oracle -e docker) on a pr_diff task emitted with this branch's verifier baked into its image (a real pallets/click commit as the oracle):
  • --ve R2E_JUDGE_ENDPOINT=http://host.docker.internal:8000/v1 --ve R2E_JUDGE_MODEL=Qwen/Qwen3.5-4B → reward-details.json: llm_judge: 1.0, judge_status: "ok", judge_model: "Qwen/Qwen3.5-4B", judge_endpoint: "http://host.docker.internal:8000/v1", final_reward: 1.0; 47 s job runtime
  • control run with no judge env vars → llm_judge: null, judge_status: "no_api_key", judge_endpoint: null, weights renormalized, final_reward: 1.0 — the default path is unchanged
  • --ve R2E_JUDGE_ENDPOINT=http://host.docker.internal:9/v1 (nothing listening) → judge_status: "network", judge weight renormalized

Out of scope

• The bare-Linux-daemon case for reaching the host (--add-host=host.docker.internal:host-gateway) is documented in the example but was not exercised; the end-to-end run below was on Docker Desktop (WSL2), where host.docker.internal resolves out of the box.
• Pre-existing drift in the same doc block: REWARD_SCHEMA.md and pr_diff.md still describe reward.json with reward / capped fields, while the verifier writes reward-details.json with final_reward and no capped (since pr_diff verifier: mirror #75 — reward-details.json sidecar #76). Separate fix.
• Release-notes entry, left for the release cut as with cli: add --llm-endpoint/--llm-key-env, keyless self-hosted LLMs #104.
• Existing published datasets keep the verifier they were emitted with; only newly generated tasks pick this up.

Closes #134

[KNambiarDJsc]

Reviewed and tested 140fb59:

• Unit tests: tests/test_pr_diff_verifier.py + tests/test_pipeline_pr_diff.py: 78 passed. ruff check / ruff format --check: clean. Coverage is genuinely thorough — including the two failure modes I'd have gone looking for myself: an Anthropic-shaped reply arriving on the OpenAI route (status="parse", not a crash), and a thinking-mode server returning content: null with the real text elsewhere (also "parse", not a TypeError).
• Real end-to-end run, not just the mocks: I stood up a real local HTTP server speaking the OpenAI /chat/completions shape and ran the actual verifier main() against it — same code path a Harbor container runs, R2E_JUDGE_ENDPOINT/R2E_JUDGE_MODEL passed exactly as --ve would:

  {
    "final_reward": 0.915,
    "components": {..., "llm_judge": 0.83},
    "judge_model": "local-test-model",
    "judge_endpoint": "http://127.0.0.1:8791/v1",
    "judge_status": "ok"
  }

  The judge score my fake server returned flowed correctly into the weighted reward.
• The "never forward ANTHROPIC_API_KEY to a custom endpoint" behavior is correct and matches the pattern #104 established for the main LLM client — good consistency across the codebase.
• test_verifier_source_is_stdlib_only is a nice addition independent of this PR's main change — it locks in an invariant (the verifier is base64-baked into a bare python:3.12-slim image, so it can never depend on anything beyond stdlib) that nothing was previously testing directly.

LGTM. Two small, non-blocking things:

1. docs/pipelines/README.md and pr_diff.md both use http://host.docker.internal:8000/v1 in the self-hosted example. That's correct for Docker Desktop, and the comment does call out the Linux bare-daemon caveat (--add-host=host.docker.internal:host-gateway) — just flagging that a reader skimming the command block rather than the prose above it could copy-paste the Docker Desktop form on a Linux CI runner and get a connection refused. Not asking for a change, just noting it since it's the one part of the docs that's environment-dependent.
2. _judge_config_from_env's docstring says R2E_JUDGE_MODEL is "then required" when an endpoint is set, but returns "" rather than raising when it's missing — llm_judge catches that downstream via the new no_judge_model status. Accurate as implemented (fails closed, doesn't crash), the docstring phrasing just reads slightly stronger than what actually happens at that specific function boundary.

[surajsharan]

@KNambiarDJsc Thanks for the careful pass, and for running it against a real OpenAI-shaped server, that's the same path the Harbor container takes, so it's good to see the score flow through the weights independently.

Both points taken, pushed as a follow-up commit:

1. You're right that the Linux caveat was easy to miss, and on a second look it was also wrong: --add-host is a docker run flag, and under harbor run nobody controls the container invocation. The example now says next to the command that host.docker.internal is a Docker Desktop name (macOS / Windows / WSL2) and gives the bare-daemon form, the host's LAN IP with the server bound to 0.0.0.0. Same note in pipelines/README.md and the R2E_JUDGE_ENDPOINT row in ENV.md.

2. Reworded the _judge_config_from_env docstring to describe what happens at that boundary: an unset R2E_JUDGE_MODEL on the endpoint route comes back as "" and llm_judge reports no_judge_model; nothing raises.

[adithya-s-k]

Thanks, tested and merged. Also checked that the custom endpoint doesn't receive the Anthropic key. Existing tasks will need their baked verifier updated to use this.