Skip to content

[Ready for Review] Adapter: DiscoverPhysics - #9

Merged
audreina17 merged 12 commits into
harbor-framework:mainfrom
Randl:discoverphysics-adapter
Nov 10, 2026
Merged

audreina17 merged 12 commits into
harbor-framework:mainfrom
Randl:discoverphysics-adapter

Conversation

@Randl

@Randl Randl commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Continues the DiscoverPhysics adapter from harbor-framework/harbor#2974, following the migration request.

DiscoverPhysics evaluates an agent's ability to discover physical laws using trajectory MSE and judged explanations. This package generates 22 tasks: 11 public N-body worlds × two seeds. The field-only wave world is excluded and documented.

The adapter now lives at src/discoverphysics/ as an independent uv project. Harbor 0.22.0 is an external optional dependency; the custom parity agent and protocol helper are installed inside the package. Tests, run configurations, and reproduction commands use the new layout. Uses the repository's standard publishing workflow; no custom packaging/evidence utilities or migration check logs are included.

Validation:

  • 137 unit tests pass; Ruff lint/format and package type checking pass.
  • Source/wheel builds pass; the installed wheel imports the custom agent and contains its templates outside either checkout.
  • At the pre-budget-setting revision 8f32585, all 22 regenerated task digests matched the September 30 oracle snapshot. The optional judge-budget setting changes two generated files per task; task instructions, oracle outputs, task limits and verifier scripts remain byte-identical. Historical validation is tied to its archived task revision.
  • Existing judged oracle: 22/22, zero errors/retries, 12m 6s, mean explanation score 0.95. Only the latest oracle run's per-task outcomes, original sanitized summary/configuration, and provenance are included under validation/.
  • Structural validation after evidence-link reconciliation: 31 passed, 0 errors, 0 warnings. parity_experiment.json records the two historical GLM configuration strata separately.

Outstanding items carried forward:

  • The recorded historical GLM subset comparisons cover Gravity (two tasks, three repetitions per side) and separate Yukawa/Oscillator/Hubble configurations (six tasks, three repetitions per side). Their run-score ranges overlap within each configuration; they are not one pooled full-suite result. Selection was retrospective and the scope limits remain documented. The selected original-side sources, task assets and portable reproduction instructions are published at a pinned revision. The portable launcher was checked offline, not replayed as a new model experiment.
  • The November 9 standard CLI compatibility check used OpenCode 1.19.5 + GLM-5.3-Flash on Gravity seed 0 and Hubble seed 0. This bounded smoke check explicitly overrode the agent timeout to 3,600 seconds; the generated tasks default to 7,200 seconds. Gravity wrote a final submission but exited nonzero; verification then encountered service errors and produced no reward. Hubble reached the smoke check’s one-hour override before writing discovered_law.py, and verification produced reward 0. Its discovery API calls completed normally; most tool time went into agent-written local numerical fitting. This is not evidence of failure under the task’s default two-hour budget. Both trials remain recorded as errors, so this check does not establish a clean CLI pass or new parity. Service diagnostics and CLI exit attribution need follow-up before compatibility sign-off.
  • Dataset PR #259 / evaluated snapshot. Requested initial registry tag: v1.0; publication remains pending.
  • Complete sanitized oracle logs, configs, experiments, and exact tasks are uploaded in the open HF oracle PR #271. The open parity PR #270 is stacked on #271: merge #271 first, then #270. Both evidence links are listed in parity_experiment.json. At the audited 43b33575 evidence snapshot, remote verification matched all 1,216 file blobs, including 673 oracle files, and 157 parity LFS hashes against the reviewed local bundle. These uploads do not imply registry publication. Existing public oracle summaries and provenance remain in validation/.
  • Additional selected evidence/source publication is blocked by HF's file-count limit: shared main already contains 1,347,979 files before DiscoverPhysics; the intended update adds 345 net files to the published PR. Git LFS uploaded successfully, but both Git and HTTP new-file publication were rejected above the 1,000,000-file limit. An existing README replacement succeeded via HTTP and links the pinned reproduction package. No shared benchmark assets were deleted or history rewritten; this needs a file-count exception or an agreed storage arrangement. Adapter-code merge and HF evidence merge remain separate operations.
  • The pinned fractional explanation-reference inconsistency and missing Coulomb Easy final-fitting training handoff are documented, with links to upstream Prompt mentions dt>0.01 but reference solution doesn't work with it SampsonML/DiscoverPhysics#2 Fix fractional-world decay grading reference SampsonML/DiscoverPhysics#3 Coulomb Easy omits training trajectories from final parameter fitting SampsonML/DiscoverPhysics#4 . Existing grading versions and scored outcomes are preserved; null fitting-loss causes remain under investigation.
  • Open original review threads: verifier credential, parity-prompt access, and experiment work bound. Shared verifier-log visibility is also documented as an outstanding follow-up. These are not marked resolved by migration.

Migration and validation were prepared with Codex assistance.

Judge output-budget follow-up

A frozen-output replay reproduced a judge failure mode: GLM-5.3-Flash used the 1,024-token upstream budget on reasoning, with finish_reason=length and no complete score. The original failed run lacked internal diagnostics, so its precise cause remains unconfirmed. Commit cb0fa46 adds the trusted service-only DISCOVERPHYSICS_JUDGE_MAX_TOKENS setting; unset retains upstream's default. The same frozen submission graded successfully in three of three verifier-only trials at 8,192 tokens, all finishing normally. These controls are not new CLI, oracle-suite or parity results, and no historical score was replaced.

Validation: 137 tests; lint, format and type checks; wheel/sdist builds; clean-source structural validation 31 passed/0 errors/0 warnings; all 22 tasks regenerated twice identically, with only Compose judge configuration and evaluation bridge changing from the previous revision.

Fresh stock OpenCode checks on the unchanged 8f32585 Gravity/Hubble task assets are continuing with Harbor 0.25.0 and the actual default 7,200-second agent budget. Those already-running jobs retain their original 1,024-token judge setting; their outcomes will be reported separately.

@audreina17

Copy link
Copy Markdown
Collaborator

/review-adapter

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Structural validation of commit b12ae1b at 2026-10-27 02:22:59 UTC

Adapter contract v1 (docs/adapters.mdx).

✅ Adapter Validation: discoverphysics

✅ 31 passed | ❌ 0 errors | ⚠️ 0 warnings

Passed checks (31)
  • ✅ README.md exists
  • ✅ parity_experiment.json exists
  • ✅ adapter_metadata.json exists
  • ✅ src/discoverphysics/ package exists
  • ✅ src/discoverphysics/adapter.py exists
  • ✅ src/discoverphysics/main.py exists
  • ✅ src/discoverphysics/task-template/ directory exists
  • ✅ src/discoverphysics/task-template/task.toml exists
  • ✅ src/discoverphysics/task-template/instruction.md exists
  • ✅ src/discoverphysics/task-template/environment/Dockerfile exists
  • ✅ src/discoverphysics/task-template/tests/test.sh exists
  • ✅ src/discoverphysics/task-template/solution/solve.sh exists
  • ✅ Template [task].name present
  • ✅ Template [task].authors present
  • ✅ parity_experiment.json is valid JSON array
  • ✅ Parity PR links are non-empty with valid URLs
  • ✅ adapter_metadata.json is valid JSON array
  • ✅ README section Overview present
  • ✅ README section What is present
  • ✅ README section Adapter Features present
  • ✅ README section Generated Task Structure present
  • ✅ README section Run Evaluation present
  • ✅ README section Usage present
  • ✅ README section Parity present
  • ✅ README section Notes & Caveats present
  • ✅ README section Installation / Prerequisites present
  • ✅ README section Citation present
  • ✅ README section Authors & Contributions present
  • ✅ Parity table column count correct
  • ✅ test.sh writes to reward path
  • ✅ No canary strings found

@audreina17

Copy link
Copy Markdown
Collaborator

@Randl could you upload the prepared, sanitized archive for your September 30
oracle run (22/22) to adapters/discoverphysics/oracle/ in
harborframework/parity-experiments
and link the Hugging Face PR here? The PR description says the archive already
contains the logs, configs, experiments, exact tasks, and checksums.

We'll handle a separate upload of our partial GLM parity evidence, with its
tested scope and qualifications documented. This request is for publishing your
existing oracle evidence; no duplicate oracle run is needed for our parity upload.

@Randl

Randl commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@audreina17

audreina17 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

@Randl Thanks for uploading the 22/22 oracle evidence. Partial parity reached Harbor's run-score range-overlap criterion on these tested GLM subsets, with three repetitions per side:

  • Gravity stream-v1: seeds 0/1, 2 tasks. Upstream and Harbor both scored 1.0000 ± 0.0000.
  • Durable-v3: Yukawa seeds 2/3, Oscillator seeds 0/1, and Hubble seeds 0/1, 6 tasks. Upstream scored 0.1111 ± 0.0556; Harbor scored 0.1667 ± 0.0000.

Scores are fractions, reported as mean ± sample SEM. Both discovery and judging used zai-org/GLM-5.3-Flash. The two runtime configurations are reported separately.

The metadata contribution is ready for review at Randl#1 and targets your discoverphysics-adapter branch; merging it updates this PR. It changes only the parity JSON, README summary, and adapter metadata. All 31 structural checks pass with no errors or warnings. HF parity PR #270 contains the supporting evidence and is stacked on your oracle PR #271.

This establishes the range-overlap result for the listed historical subsets, not full-suite acceptance or migrated-head equivalence. The reviewer-runtime patch/fork is deferred. Could you review and merge the metadata contribution? The existing security/fidelity findings and standard CLI-agent check remain outstanding.

@Randl Randl changed the title [WIP] Adapter: DiscoverPhysics [Ready for Review] Adapter: DiscoverPhysics Oct 23, 2026
@Randl
Randl marked this pull request as ready for review October 23, 2026 06:05
@crystalxyz

Copy link
Copy Markdown
Collaborator

/review-adapter

@github-actions

Copy link
Copy Markdown
Contributor

Adapter review of commit b12ae1b (applies only to this revision).

Adapter review: discoverphysics (PR #9 @ b12ae1b401695c0040454d436a39220f911d3654)

Contract: Adapter contract v1, review base revision c6eb509c459c994b90750d7d880dc93530e55d8f.
Scope: Static review of reviewed-pr/src/discoverphysics/ only. I did not execute any code or start any runs. I fetched the PR links and the pinned upstream agent.py file to check them.
Pins: upstream SampsonML/DiscoverPhysics@33b7fa9d. The default dataset is 11 n-body worlds × seeds 0/1 = 22 tasks.

Verdict: Not ready to accept. Parity evidence is the blocker.

The adapter code, the sandboxing and the oracle are high quality. The parity record does not meet ADP-PARITY or the review guidance:

  • It covers 2/22 tasks on the default dataset and 6/44 tasks on a non-default variant.
  • The agent and runtime used for parity are not in this PR, and there is no original-side repo.
  • Some worlds and models were excluded after results were seen, and the README does not say so.
  • The required standard CLI-agent run has not been done.

The authors openly call this "partial historical evidence" and do not claim acceptance. That disclosure is good, but it does not satisfy the requirement.


Status of previously raised issues

  • Earlier structural error "pending parity_experiment.json": resolved. The file is now present and is valid against the schema shape.
  • Prior bot comment (github-actions structural validation, 31 passed, 0 errors): no issues to re-check.
  • Open threads from harbor#2974 (verifier credential in the main container, parity-prompt access, no per-round experiment work bound, judge reasoning in the shared verifier log): still present. README Notes & Caveats now discloses them as open limitations (README.md:359-362). None is fixed in code:
    • physics_server.py:_experiment still has no work bound.
    • evaluation_bridge.py:244 still emits judge_reasoning.
    • The result token still lives on a read-only mount in main.

1. Adapter code layout and logic — PASS (minor notes)

  • ✅ src/discoverphysics/{adapter.py,main.py,__init__.py,task-template/} are all present. __init__.py is __all__ = [].
  • ✅ The template has task.toml, instruction.md, environment/Dockerfile, solution/solve.sh and tests/test.sh.
  • ✅ main.py supports --output-dir, --limit, --overwrite and --task-ids, and adds --num-seeds, --upstream-root, --repo-url and --upstream-commit. It imports DiscoverPhysicsAdapter from .adapter and calls .run().
  • ✅ Class name DiscoverPhysicsAdapter has run(self) and writes under self.output_dir.
  • ✅ pyproject.toml: name = "harbor-discoverphysics-adapter", and [project.scripts] discoverphysics = "discoverphysics.main:main".
  • ✅ Default output is datasets/discoverphysics.
  • ✅ Error handling is solid:
    • git failures are wrapped.
    • Upstream revision and cleanliness are checked.
    • --overwrite does an atomic staging-and-rollback replacement (adapter.py:191-223).
    • Unknown task IDs are rejected.
  • ✅ pathlib is used throughout and there are no bare except.
  • ✅ Task IDs (<world>__seed-<n>) are deterministic, registry-safe, and use a fixed world order.
  • ℹ️ run() clones upstream only to validate it. Nothing from the clone is copied into tasks, because the physics image fetches the same commit at build time. This works but means a network clone on every generation. Consider skipping the clone when --upstream-root is not given and only the template is needed. Optional.
  • ℹ️ main.py:22 hardcodes default=2 instead of reusing DEFAULT_NUM_SEEDS. This is cosmetic.
  • ✅ The README documents special treatments: exclusions of wave, force_geography and running_coupling, the oracle dt, the coulomb_easy fit-world omission, and the E@k denominator.

2. README.md — PARTIAL

  • ✅ All required sections are present in the required order, followed by an Acknowledgement section (ADP-README).
  • ✅ Usage uses uv run discoverphysics ....
  • ✅ The overview gives the task count and explains the exclusions.
  • ✅ Numbers match parity_experiment.json: 3 runs, 2/22 and 6/44.
  • ⚠️ Inconsistent publication claims. README.md:352-354 says oracle evidence is "published in HF oracle PR #271". validation/README.md:18 says "Hugging Face publication is pending". HF #271 is open and unmerged. Pick one wording and say "submitted (open)".
  • ⚠️ The registry commands (README.md:117-119) cannot be run until harbor-datasets #259 merges. This is disclosed.
  • ⚠️ Undisclosed exclusions (ADP-README, Notes & Caveats). The HF #270 description says "Coulomb and DeepSeek were excluded retrospectively at the reviewer's request". It also says Ether's strict first run is incomplete and the "Ether feedback discrepancy remains unresolved." The adapter README does not mention Ether, Coulomb or DeepSeek parity attempts or why they were dropped. Benchmark deviations and exclusions must be listed under Notes & Caveats.
  • ℹ️ The content is dense and very hedged, but it is accurate and readable.

3. task-template files — PASS

  • ✅ [task].name = "discoverphysics/{task_id}".
  • ✅ authors lists all 7 paper authors, with email for the first (ADP-AUTHORS).
  • ✅ No canary strings and no terminal-bench or harbor comments.
  • ✅ test.sh removes stale reward files and then runs python3 -I /tests/verify.py. verify.py writes reward.txt and reward.json on every scored path.
  • ✅ Timeouts and resources (agent 7200 s, verifier 2400 s, 4 CPU / 8 GB) suit a 16-round, LLM-judged task.
  • ✅ The Dockerfiles pin base images by digest and pin dependencies.
  • ℹ️ task.toml has schema_version = "1.3" plus artifacts, while [task].version = "1.0". The spec example uses top-level version = "1.0". Confirm this schema is accepted by the Harbor 0.22.0 runtime you pinned. The oracle run suggests it is.
  • ℹ️ An infrastructure failure (503s exhausted, or a non-400 HTTP error) exits 1 without a reward file. This is intentional, so infra errors are not scored as zero, and it is documented in Troubleshooting.

4. parity_experiment.json — FAIL

  • ✅ number_of_runs (3) equals the length of each *_runs array.
  • ✅ SEM recomputed (ADP-STATS):
    • Gravity: [1,1,1] vs [1,1,1] gives 1.0 ± 0 on both sides. ✓
    • Durable-v3 original [1/6, 1/6, 0]: mean 0.11111, sample SEM = sqrt(0.018519/6) = 0.055556. ✓
    • Durable-v3 Harbor [1/6]×3: 0.16667 ± 0. ✓
    • The README table matches the JSON.
  • ✅ Range overlap (ADP-PARITY):
    • Gravity [1,1] vs [1,1]: overlaps.
    • Durable-v3 [0, 0.1667] vs [0.1667, 0.1667]: overlaps only at the boundary value 0.1667.
  • ❌ original_parity_repo is "". This field is required. Without it, nobody can reproduce the original side of the recorded runs.
  • ❌ The recorded agent is not in this PR. agent is DiscoverPhysicsOpenAIParityAgent@sha256:... (two different hashes for the two strata). The only parity agent shipped is DiscoverPhysicsParityAgent (Anthropic-only, discoverphysics_parity_agent.py:18). The README also says the runtime had "reviewer changes" and the "patch/fork is deferred". So the code that produced both sides of the parity numbers cannot be reproduced from this PR.
  • ❌ Second entry is not on the default dataset. It reports adapted_benchmark_size: 44, a historical four-seed variant, and includes Yukawa seeds 2/3, which are not in the registered 22-task dataset. The README does not explain why Yukawa uses seeds 2/3 while the other worlds use 0/1.
  • ⚠️ The judge model differs from the shipped configuration. Parity used GLM-5.3-Flash as judge, while tasks default to claude-opus-4-6. The parity runs therefore did not test the grading setup the dataset ships with.
  • ✅ The URL formats are valid. dataset_pr uses harbor-framework/harbor-datasets, which resolves and is the correct PR.

5. adapter_metadata.json — PARTIAL

  • ✅ adapter_builders lists the adapter author (Evgenii Zheltonozhskii), not the benchmark authors.
  • ✅ Sizes agree with parity_experiment.json: 22/2 and 44/6, with sampling rates 0.0909 and 0.1364.
  • ⚠️ ADP-COST: parity_costs: "Unknown (historical spend not reconstructed)" is a string but not a canonical USD value. This is a warning, not a blocker; give an estimate or use null.
  • ⚠️ parity_matching_agents: [] and parity_unmatching_agents: [] are empty. The spec expects agent@version+model entries or ["None"]. Leaving them empty does fit the authors' position that acceptance has not been established.
  • ⚠️ registry_benchmark_size: 0 reflects the pending registry state. Update it to 22 after publishing.
  • ⚠️ A second split, public-worlds-four-seed-reviewer-variant (44 tasks), is listed although it is not a published dataset. Consider moving it into notes so it is not read as a supported split.

6. Parity verification — FAIL (blocking)

  • ❌ Coverage. Parity covers 2/22 tasks on the shipped dataset; the other 6 tasks are from a different universe. The README gives no selection rationale. HF #270 shows the subset is what remained after Coulomb, DeepSeek and Ether were dropped after results were seen. The review guidance says not to choose favorable subsets after observing outcomes, and to keep incomplete or infra attempts in a ledger. Disclose all attempted worlds and models, with results and exclusion reasons, in README and JSON notes. Or run a pre-registered parity subset.
  • ❌ Original-side reproduction is missing. The README reproduction commands (parity/original_*.yaml, run_parity*.yaml) are for an Opus plan that has not been run. They do not reproduce the recorded GLM results (README.md:274-278 says so). There is no fork link.
  • ❌ Custom agent without a standard CLI-agent run. This is a Scenario 3 adapter and requires a separate run with a standard agent such as codex or claude-code. The README (README.md:337-339, 356-357) says this is pending.
  • ⚠️ The agent versions use a SHA-256 of the agent source rather than a semantic @version. That is acceptable for pinning, but it does not match the shipped agent's version() of 0.1.0.
  • ⚠️ Low discriminating power:
    • Gravity is at ceiling, 1.0 on every run on both sides.
    • Oscillator fails in every run on both sides (README.md:261-262).
    • Durable-v3 overlaps only at one shared value.
    • Even with these runs, there is little evidence that the two sides measure the same thing across the benchmark.
  • Next step needed: follow the plan the README already outlines:
    1. Get team approval.
    2. Run the six-task sanity check on both sides.
    3. Run one full 22-task repetition on both sides.
    4. Run three repetitions, with a published original-side fork or patch.
    5. Run one standard CLI agent on the Harbor side.

7. Oracle verification — PASS (with caveat)

  • ✅ The judged oracle run discoverphysics-oracle-20260930-102507 passed all 22 tasks (22/22). Evidence: validation/, recomputed-metrics.json with trial_pass_rate 1.0 and mean explanation 0.95, and HF #271 (open).
  • ✅ Oracle results are kept separate from parity.
  • ⚠️ The oracle laws integrate with dt=0.005. The pinned upstream agent.py _run_policy_note() tells agents "ALWAYS set dt > 0.005" and "the SMALLEST value of dt you are allowed to use is 0.01". So the oracle uses a value the agent prompt forbids.
    • The README says that at dt=0.01 the canonical laws fail coulomb_easy (nMSE 3.61) and ether (0.374).
    • This is disclosed and comes from upstream; the code never enforces dt. Reviewers should still note that the oracle shows the harness works, not that a prompt-compliant agent can solve those two worlds.
  • ⚠️ The Harbor runtime revision used for the oracle run was not recorded. This is disclosed.

8. Link verification — PARTIAL

Links I fetched:

Link Status
adapters#9 ✅ Correct PR ("[Ready for Review] Adapter: DiscoverPhysics"), open
harbor#2974 ✅ Original WIP, closed, moved to #9
harbor-datasets#259 ✅ "Add DiscoverPhysics: 11 N-body worlds at two seeds", open
HF parity #270 ✅ DiscoverPhysics partial GLM parity, open, stacked on #271
HF oracle #271 ✅ DiscoverPhysics 22/22 oracle evidence, open
Upstream agent.py at the pinned commit ✅ Resolves; its timestep and min-round semantics match the adapter

9. PR completeness — PASS

  • adapter_pr lists adapters#9 and harbor#2974.
  • dataset_pr lists #259.
  • parity_pr lists HF #270.
  • ⚠️ Add HF #271 (oracle evidence) to parity_pr, or to metadata notes, since #270 is stacked on it.

10. Task generation verification — PASS

  • All placeholders are substituted in task.toml, docker-compose.yaml and the physics-server/Dockerfile: {task_id}, {world}, {seed}, {seed_pool_size}, {noise_std}, {world_variance}, {world_index}, {upstream_commit}, {upstream_repository}.
  • upstream_repository is JSON-quoted for the Dockerfile ARG.
  • tests/task_manifest.json and the oracle files under solution/ are written for each task.
  • Execute bits are set.
  • I did not compare against a generated dataset, because none is in this checkout. The README says the task snapshots match harbor-datasets@4952f1fe.

11. Oracle smoke test — PASS

  • solve.sh runs two experiments, satisfying upstream's two-round minimum, then copies the canonical law and explanation into /workspace.
  • verify.py:
    1. Exports experiments.
    2. Rejects symlinks and non-regular files.
    3. POSTs to physics:/verify with the result token.
    4. Writes the reward.
  • The grading logic (evaluation_bridge.py:229-234) uses normalized_mse = mean_pos_error / WORLD_VARIANCE, with reward = (< 0.1 AND explanation >= 0.75). This matches the stated upstream rubric.
  • The fit-world set omits coulomb_easy, as in pinned upstream. This is disclosed, and upstream issue Support AA-LCR 1.1 with pinned data and updated grading #4 is filed.
  • The physics image installs everything the server, bridge and judge need. The main image installs anthropic for parity_complete.py.

12. Trust check — PASS (code) / CONCERN (evidence)

The implementation is careful and convincing:

  • Sidecar isolation, an internal-only network for the law runner, a read-only root filesystem, dropped capabilities and PID and memory limits.
  • Upstream fetched by pinned commit and verified.
  • Untrusted law code runs remotely in the law runner, with upstream's _compile_law replaced.

The concern is on the evidence side: the parity numbers came from code that is not in the PR, and some subsets were excluded after results were known (see §6).

13. Benchmark vulnerability check

13a Leakage — PASS

  • solution/ and tests/ are not copied into main.
  • Held-out cases, true laws, rubrics and ideal explanations exist only in the physics image.
  • The law runner has no simulator source.
  • Rounded observations go to the agent; full-precision data is exported only to /logs/verifier after discovery.

13b Identity leakage — MINOR FAIL

  • Agent-visible paths name the benchmark:
    • /run/discoverphysics-agent/token, referenced in /app/discover.py:13.
    • /run/discoverphysics-result.
  • The physics service exposes /parity-prompt, which returns the native upstream system prompt.
  • The mission prompt mentions a "native chat harness".
  • Suggestion: rename the mounts and paths to neutral names such as /run/lab-agent/token.
  • instruction.md itself is clean.

13c — N/A (no repository checkout).

13d Evaluation integrity — PASS with disclosed residual risk

  • Grading happens in the physics sidecar, gated by a result token in a root-only 0700 directory and 0600 file. The agent runs as agent (uid 1000).
  • test.sh deletes and rewrites the reward files.
  • It uses python3 -I, and /usr/local/bin is not writable by the agent.
  • The verify endpoint caches the result per submission and rejects changed submissions.
  • Residual risks:
    • The result token shares the main container with the agent. Harbor must keep the agent at the image's USER agent, and the README says protection depends on that.
    • An agent-spawned background process could keep running into the verification phase. It cannot reach the token, but you should confirm /logs/verifier is not writable by uid 1000 after test.sh runs.
    • Token comparison uses !=, which is not constant-time. This is low risk on an internal network.

13e Reward hacking — PASS with a semantic note

  • There is no skip variable, and the instructions never mention reward files.
  • ⚠️ Upstream rejects a <final_law> before round 2, and the parity agent mirrors that. For terminal and CLI agents, though, the verifier does not enforce the minimum rounds; instruction.md only says "should". A CLI agent could submit after 0 experiments. This differs from upstream semantics for non-parity agents. It is disclosed in README.md:395-397, but consider enforcing rounds_used >= MIN_ROUNDS in /verify to match upstream acceptance.
  • ⚠️ There is no per-request work bound on /experiment (open thread). An agent could submit very large experiments that stall the shared physics service. That is availability, not integrity.

Summary

Blocking:

  1. ADP-PARITY — parity evidence.
    • It covers 2 of the 22 shipped tasks.
    • Its second stratum is from a non-default 44-task variant.
    • The agent and runtime that produced it are not in the PR.
    • original_parity_repo is empty.
    • The world and model subset was chosen after results were seen.
  2. The required standard CLI-agent run is missing for a custom-agent (Scenario 3) adapter.
  3. ADP-README — Notes & Caveats does not disclose the Ether, Coulomb and DeepSeek parity attempts and why they were excluded.

Disclosed limitations (acceptable as documented):

  • wave, force_geography and running_coupling are excluded.
  • The oracle uses dt=0.005.
  • E@k is capped at 50% by upstream's fixed 22-world denominator.
  • coulomb_easy gets no training trajectories for the final fit.
  • The verifier token shares main with the agent.
  • Judge reasoning is written to the verifier log.

Non-blocking fixes:

  • Make the HF publication wording consistent (#271 is open, not published).
  • Add #271 to parity_pr.
  • Use a canonical USD value for parity_costs (ADP-COST).
  • Use ["None"] or explicit entries for the matching and unmatching agent lists.
  • Rename the agent-visible paths that contain discoverphysics.
  • Consider enforcing upstream's minimum rounds in /verify.
  • Record the Harbor revision used in future runs.

Not verified by me:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants