Repository navigation
[Ready for Review] Adapter: DiscoverPhysics - #9
Conversation
|
/review-adapter |
|
Structural validation of commit b12ae1b at 2026-10-27 02:22:59 UTC Adapter contract v1 ( ✅ Adapter Validation:
|
|
@Randl could you upload the prepared, sanitized archive for your September 30 We'll handle a separate upload of our partial GLM parity evidence, with its |
|
@Randl Thanks for uploading the 22/22 oracle evidence. Partial parity reached Harbor's run-score range-overlap criterion on these tested GLM subsets, with three repetitions per side:
Scores are fractions, reported as mean ± sample SEM. Both discovery and judging used The metadata contribution is ready for review at Randl#1 and targets your This establishes the range-overlap result for the listed historical subsets, not full-suite acceptance or migrated-head equivalence. The reviewer-runtime patch/fork is deferred. Could you review and merge the metadata contribution? The existing security/fidelity findings and standard CLI-agent check remain outstanding. |
Record DiscoverPhysics partial historical parity evidence
|
/review-adapter |
|
Adapter review of commit b12ae1b (applies only to this revision). Adapter review:
|
| Link | Status |
|---|---|
| adapters#9 | ✅ Correct PR ("[Ready for Review] Adapter: DiscoverPhysics"), open |
| harbor#2974 | ✅ Original WIP, closed, moved to #9 |
| harbor-datasets#259 | ✅ "Add DiscoverPhysics: 11 N-body worlds at two seeds", open |
| HF parity #270 | ✅ DiscoverPhysics partial GLM parity, open, stacked on #271 |
| HF oracle #271 | ✅ DiscoverPhysics 22/22 oracle evidence, open |
Upstream agent.py at the pinned commit |
✅ Resolves; its timestep and min-round semantics match the adapter |
- Not individually fetched: upstream PR Align adapter CI and documentation around a versioned contract #3, upstream issue Support AA-LCR 1.1 with pinned data and updated grading #4, the Randl fork snapshot link and the arXiv reference.
⚠️ validation/README.md:19and README.md:351 use repo-relative links (../../docs/adapters.mdx,../../../skills/...). These resolve in-repo but not on registry pages.
9. PR completeness — PASS
adapter_prlists adapters#9 and harbor#2974.dataset_prlists #259.parity_prlists HF #270.⚠️ Add HF #271 (oracle evidence) toparity_pr, or to metadata notes, since #270 is stacked on it.
10. Task generation verification — PASS
- All placeholders are substituted in
task.toml,docker-compose.yamland thephysics-server/Dockerfile:{task_id},{world},{seed},{seed_pool_size},{noise_std},{world_variance},{world_index},{upstream_commit},{upstream_repository}. upstream_repositoryis JSON-quoted for the DockerfileARG.tests/task_manifest.jsonand the oracle files undersolution/are written for each task.- Execute bits are set.
- I did not compare against a generated dataset, because none is in this checkout. The README says the task snapshots match harbor-datasets@
4952f1fe.
11. Oracle smoke test — PASS
solve.shruns two experiments, satisfying upstream's two-round minimum, then copies the canonical law and explanation into/workspace.verify.py:- Exports experiments.
- Rejects symlinks and non-regular files.
- POSTs to
physics:/verifywith the result token. - Writes the reward.
- The grading logic (evaluation_bridge.py:229-234) uses
normalized_mse = mean_pos_error / WORLD_VARIANCE, with reward = (< 0.1AND explanation>= 0.75). This matches the stated upstream rubric. - The fit-world set omits
coulomb_easy, as in pinned upstream. This is disclosed, and upstream issue Support AA-LCR 1.1 with pinned data and updated grading #4 is filed. - The physics image installs everything the server, bridge and judge need. The
mainimage installsanthropicforparity_complete.py.
12. Trust check — PASS (code) / CONCERN (evidence)
The implementation is careful and convincing:
- Sidecar isolation, an internal-only network for the law runner, a read-only root filesystem, dropped capabilities and PID and memory limits.
- Upstream fetched by pinned commit and verified.
- Untrusted law code runs remotely in the law runner, with upstream's
_compile_lawreplaced.
The concern is on the evidence side: the parity numbers came from code that is not in the PR, and some subsets were excluded after results were known (see §6).
13. Benchmark vulnerability check
13a Leakage — PASS
solution/andtests/are not copied intomain.- Held-out cases, true laws, rubrics and ideal explanations exist only in the
physicsimage. - The law runner has no simulator source.
- Rounded observations go to the agent; full-precision data is exported only to
/logs/verifierafter discovery.
13b Identity leakage — MINOR FAIL
- Agent-visible paths name the benchmark:
/run/discoverphysics-agent/token, referenced in/app/discover.py:13./run/discoverphysics-result.
- The physics service exposes
/parity-prompt, which returns the native upstream system prompt. - The
missionprompt mentions a "native chat harness". - Suggestion: rename the mounts and paths to neutral names such as
/run/lab-agent/token. instruction.mditself is clean.
13c — N/A (no repository checkout).
13d Evaluation integrity — PASS with disclosed residual risk
- Grading happens in the
physicssidecar, gated by a result token in a root-only 0700 directory and 0600 file. The agent runs asagent(uid 1000). test.shdeletes and rewrites the reward files.- It uses
python3 -I, and/usr/local/binis not writable by the agent. - The verify endpoint caches the result per submission and rejects changed submissions.
- Residual risks:
- The result token shares the
maincontainer with the agent. Harbor must keep the agent at the image'sUSER agent, and the README says protection depends on that. - An agent-spawned background process could keep running into the verification phase. It cannot reach the token, but you should confirm
/logs/verifieris not writable by uid 1000 aftertest.shruns. - Token comparison uses
!=, which is not constant-time. This is low risk on an internal network.
- The result token shares the
13e Reward hacking — PASS with a semantic note
- There is no skip variable, and the instructions never mention reward files.
⚠️ Upstream rejects a<final_law>before round 2, and the parity agent mirrors that. For terminal and CLI agents, though, the verifier does not enforce the minimum rounds;instruction.mdonly says "should". A CLI agent could submit after 0 experiments. This differs from upstream semantics for non-parity agents. It is disclosed in README.md:395-397, but consider enforcingrounds_used >= MIN_ROUNDSin/verifyto match upstream acceptance.⚠️ There is no per-request work bound on/experiment(open thread). An agent could submit very large experiments that stall the shared physics service. That is availability, not integrity.
Summary
Blocking:
- ADP-PARITY — parity evidence.
- It covers 2 of the 22 shipped tasks.
- Its second stratum is from a non-default 44-task variant.
- The agent and runtime that produced it are not in the PR.
original_parity_repois empty.- The world and model subset was chosen after results were seen.
- The required standard CLI-agent run is missing for a custom-agent (Scenario 3) adapter.
- ADP-README — Notes & Caveats does not disclose the Ether, Coulomb and DeepSeek parity attempts and why they were excluded.
Disclosed limitations (acceptable as documented):
wave,force_geographyandrunning_couplingare excluded.- The oracle uses
dt=0.005. - E@k is capped at 50% by upstream's fixed 22-world denominator.
coulomb_easygets no training trajectories for the final fit.- The verifier token shares
mainwith the agent. - Judge reasoning is written to the verifier log.
Non-blocking fixes:
- Make the HF publication wording consistent (#271 is open, not published).
- Add #271 to
parity_pr. - Use a canonical USD value for
parity_costs(ADP-COST). - Use
["None"]or explicit entries for the matching and unmatching agent lists. - Rename the agent-visible paths that contain
discoverphysics. - Consider enforcing upstream's minimum rounds in
/verify. - Record the Harbor revision used in future runs.
Not verified by me:
- Generated-task equality with harbor-datasets@
4952f1fe. - Contents of the HF #270 and #271 artifacts beyond their descriptions.
- Upstream evaluator and law-runner timeout equivalence. The bridge uses a 30 s per-call timeout to the law runner.
- Upstream PR Align adapter CI and documentation around a versioned contract #3, issue Support AA-LCR 1.1 with pinned data and updated grading #4 and the arXiv link.
Continues the DiscoverPhysics adapter from harbor-framework/harbor#2974, following the migration request.
DiscoverPhysics evaluates an agent's ability to discover physical laws using trajectory MSE and judged explanations. This package generates 22 tasks: 11 public N-body worlds × two seeds. The field-only wave world is excluded and documented.
The adapter now lives at
src/discoverphysics/as an independent uv project. Harbor 0.22.0 is an external optional dependency; the custom parity agent and protocol helper are installed inside the package. Tests, run configurations, and reproduction commands use the new layout. Uses the repository's standard publishing workflow; no custom packaging/evidence utilities or migration check logs are included.Validation:
8f32585, all 22 regenerated task digests matched the September 30 oracle snapshot. The optional judge-budget setting changes two generated files per task; task instructions, oracle outputs, task limits and verifier scripts remain byte-identical. Historical validation is tied to its archived task revision.validation/.parity_experiment.jsonrecords the two historical GLM configuration strata separately.Outstanding items carried forward:
discovered_law.py, and verification produced reward 0. Its discovery API calls completed normally; most tool time went into agent-written local numerical fitting. This is not evidence of failure under the task’s default two-hour budget. Both trials remain recorded as errors, so this check does not establish a clean CLI pass or new parity. Service diagnostics and CLI exit attribution need follow-up before compatibility sign-off.v1.0; publication remains pending.parity_experiment.json. At the audited43b33575evidence snapshot, remote verification matched all 1,216 file blobs, including 673 oracle files, and 157 parity LFS hashes against the reviewed local bundle. These uploads do not imply registry publication. Existing public oracle summaries and provenance remain invalidation/.dt>0.01but reference solution doesn't work with it SampsonML/DiscoverPhysics#2 Fix fractional-world decay grading reference SampsonML/DiscoverPhysics#3 Coulomb Easy omits training trajectories from final parameter fitting SampsonML/DiscoverPhysics#4 . Existing grading versions and scored outcomes are preserved; null fitting-loss causes remain under investigation.Migration and validation were prepared with Codex assistance.
Judge output-budget follow-up
A frozen-output replay reproduced a judge failure mode: GLM-5.3-Flash used the 1,024-token upstream budget on reasoning, with
finish_reason=lengthand no complete score. The original failed run lacked internal diagnostics, so its precise cause remains unconfirmed. Commitcb0fa46adds the trusted service-onlyDISCOVERPHYSICS_JUDGE_MAX_TOKENSsetting; unset retains upstream's default. The same frozen submission graded successfully in three of three verifier-only trials at 8,192 tokens, all finishing normally. These controls are not new CLI, oracle-suite or parity results, and no historical score was replaced.Validation: 137 tests; lint, format and type checks; wheel/sdist builds; clean-source structural validation 31 passed/0 errors/0 warnings; all 22 tasks regenerated twice identically, with only Compose judge configuration and evaluation bridge changing from the previous revision.
Fresh stock OpenCode checks on the unchanged
8f32585Gravity/Hubble task assets are continuing with Harbor 0.25.0 and the actual default 7,200-second agent budget. Those already-running jobs retain their original 1,024-token judge setting; their outcomes will be reported separately.