From c0af4634edac1503434b8df4817c9a0895e466bc Mon Sep 17 00:00:00 2001 From: audreina17 Date: Thu, 22 Oct 2026 02:35:29 -0700 Subject: [PATCH 1/3] docs(discoverphysics): record partial historical parity evidence --- src/discoverphysics/README.md | 115 +++++++++++++++------ src/discoverphysics/adapter_metadata.json | 32 +++++- src/discoverphysics/parity_experiment.json | 80 ++++++++++++++ 3 files changed, 192 insertions(+), 35 deletions(-) create mode 100644 src/discoverphysics/parity_experiment.json diff --git a/src/discoverphysics/README.md b/src/discoverphysics/README.md index 4620f581..b80ed7d2 100644 --- a/src/discoverphysics/README.md +++ b/src/discoverphysics/README.md @@ -236,28 +236,78 @@ uv run discoverphysics \ ### Parity type: Scenario 3 (custom upstream agent) -Parity is being run by Audrey Zheng, as coordinated in the -[migration request](https://github.com/harbor-framework/harbor/pull/2974#issuecomment-6004953490); -reviewed results are not yet recorded here. Migration does not launch duplicate runs. -The experiment uses the full 22-task -public suite, `DiscoveryAgent@33b7fa9d` on upstream and -`DiscoverPhysicsParityAgent@0.1.0` in Harbor, `claude-opus-4-7` as agent, -`claude-opus-4-6` as judge, critic disabled, N-body engine, noise fraction -`0.05`, seeds 0 and 1, 16 rounds, and MSE fitting enabled. Both sides use -Python 3.12 and `anthropic==0.64.0` for model calls. - -The parity plan must be approved by the Harbor team before API spend. Execution -then proceeds symmetrically in the required order: - -1. Run the fixed six-task sanity set on both sides. -2. Inspect errors and trajectories, then run one full 22-task repetition on - both sides. -3. Only if the full-run score ranges are compatible, run repetitions two and - three on both sides with the same commits and configuration. +The [partial historical parity evidence](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270) records three repetitions +per side using `zai-org/GLM-5.3-Flash` for both discovery and judging. Scores below +are fractions reported as mean ± sample SEM. The complete comparisons have +overlapping raw run-score ranges. These results do not establish full-adapter +acceptance or equivalence to the migrated adapter head. | Agent | Model | Metric | Number of Runs | Dataset Size | Original Benchmark Performance | Harbor Adapter Performance | -| --- | --- | --- | ---: | ---: | --- | --- | -| Not run | Not run | Trial pass rate | 0 | 0 | Pending | Pending | +| --- | --- | --- | ---: | --- | --- | --- | +| Reviewer parity agent, stream-v1 | zai-org/GLM-5.3-Flash | Gravity success rate | 3 | 2/22 (9.09%) | 1.00000000 ± 0.00000000 | 1.00000000 ± 0.00000000 | +| Reviewer parity agent, durable-v3 | zai-org/GLM-5.3-Flash | Complete six-task success rate | 3 | 6/44 (13.64%) | 0.11111111 ± 0.05555556 | 0.16666667 ± 0.00000000 | + +[parity_experiment.json](parity_experiment.json) records both configuration strata +separately, including exact agent SHA-256 versions and raw repetition arrays. +Gravity uses stream-v1 at seeds 0/1. Durable-v3 uses a persistent worker and a +four-seed universe: Yukawa seeds 2/3, Oscillator seeds 0/1, and Hubble seeds 0/1. +The six-task metric equally weights those tasks within each repetition; it does +not pool Gravity or Ether. The four-seed universe has 44 possible tasks and is +not the default 22-task dataset submitted for registration. + +World-level rates, in the same fraction units, remain visible below and in the +[raw evidence and attempt ledger](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270): + +| World | Upstream runs | Harbor runs | Upstream mean ± SEM | Harbor mean ± SEM | +| --- | --- | --- | --- | --- | +| Gravity | 1, 1, 1 | 1, 1, 1 | 1.00000000 ± 0.00000000 | 1.00000000 ± 0.00000000 | +| Yukawa | 0, 0, 0 | 0, 0.5, 0 | 0.00000000 ± 0.00000000 | 0.16666667 ± 0.16666667 | +| Oscillator | 0, 0, 0 | 0, 0, 0 | 0.00000000 ± 0.00000000 | 0.00000000 ± 0.00000000 | +| Hubble | 0.5, 0.5, 0 | 0.5, 0, 0.5 | 0.33333333 ± 0.16666667 | 0.33333333 ± 0.16666667 | +| Ether, strict | 0, 0.5, 0.5 | missing, 1, 0.5 | 0.33333333 ± 0.16666667 | Incomplete | +| Ether, qualified original salvage | 0, 0.5, 0.5 | 0, 1, 0.5 | 0.33333333 ± 0.16666667 | 0.50000000 ± 0.28867513 | + +Ether's strict first Harbor repetition is incomplete. The original salvaged zero +has an unresolved Harbor error-feedback discrepancy, so it appears only in the +qualified sensitivity row. It is not included in the complete six-task metric. +Scored supplemental trials are not substituted for planned outcomes. +Oscillator's all-failure agreement provides limited discriminatory evidence. + +Coulomb and DeepSeek were excluded retrospectively. All outcomes in the selected +configurations remain visible, including scored failures and empty/truncated +responses. The inventory contains 67 attempts: 59 strict scored slots, six +unscored infrastructure attempts, and two scored supplemental trials outside the +60 planned slots. Discovery responses hit the length cap 83/425 times upstream +and 84/419 times in Harbor; these counts do not establish that all discrepancies +are random. + +Historical settings: W&B Inference on Modal, requested discovery/judge caps +131072/65536, 16 rounds, nominal fitting budget 180 seconds, and OpenAI SDK +1.102.0. The model ID is not an immutable snapshot. Upstream is pinned to +`33b7fa9df96de9c35744efd181ca7e5a8dd60ad5`; the historical adapter baseline is +`059eef7a960c036760e7391cdf0da3faac59f9d9` and runtime baseline is +`3c82380859d187957cfd5cd64802b076d9779550`, with reviewer changes. Full-precision +noise and fitting/runtime differences remain qualifications. No new run or +regrading was performed for this integration. + +The reviewer-runtime patch/fork is deferred. `original_parity_repo` therefore +remains empty rather than linking a repository that cannot reproduce these +runs. The HF upload retains the recorded configurations and provenance; the +commands below describe the separate Opus reference plan and do not reproduce +the historical GLM evidence. + +### Reference Opus parity plan (not the recorded GLM results) + +The existing reference configuration targets the full 22-task public suite, +`DiscoveryAgent@33b7fa9d` upstream and `DiscoverPhysicsParityAgent@0.1.0` in +Harbor, `claude-opus-4-7` for discovery and `claude-opus-4-6` for judging, critic +disabled, N-body engine, noise fraction `0.05`, seeds 0/1, 16 rounds, MSE fitting, +Python 3.12, and `anthropic==0.64.0`. + +Before new API spend, the Harbor team must approve the plan: run the fixed +six-task sanity set on both sides, inspect errors and trajectories, run the +first full repetition, then run repetitions two and three only if the first +comparison is compatible. This metadata integration launches no new runs. Original-side reproduction from the pinned DiscoverPhysics checkout: @@ -303,7 +353,7 @@ uv run --project src/discoverphysics --extra harbor harbor run -c src/discoverph Report trial pass rate as mean ± sample SEM and retain every raw per-run score. Parity passes only if the original and Harbor raw run-score ranges overlap. Do -not populate `parity_experiment.json` until real results exist. A separate +not treat historical subset overlap as full-suite validation. A separate standard Harbor CLI-agent run is also required as Harbor-only compatibility evidence; it is not an original-versus-adapter parity claim. @@ -318,12 +368,13 @@ packaging, entry points, and paths. [evaluated snapshot](https://github.com/Randl/harbor-datasets/tree/4952f1fe2e0104d6ae5467c52444dbd908b13828/datasets/discoverphysics). Registry publication is pending. For repackaging, follow the repository's [dataset registration guide](../../docs/adapters.mdx#step-8-register-the-dataset). -- [Latest oracle evidence](validation/README.md): 22/22 passed. Full logs and - experiment artifacts are prepared for `harborframework/parity-experiments`; - the public upload link is pending. Use the repository's - [upload skill](../../skills/upload-parity-experiments/SKILL.md) for publication. -- Reviewer-coordinated parity and the separate standard CLI-agent check remain - pending. An oracle pass does not establish either result. +- [Latest oracle evidence](validation/README.md): 22/22 passed. Full sanitized + logs, configurations, experiments, and exact tasks are published in + [HF oracle PR #271](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/271). +- Partial historical parity is recorded above and in `parity_experiment.json`. + Full-adapter acceptance, current-head equivalence, and the separate standard + CLI-agent check remain pending. The reviewer-runtime patch/fork is deferred. + An oracle pass does not establish parity or CLI-agent compatibility. - Open review limitations: the root-only verifier token shares the main container with the agent; discovery credentials expose native parity prompts; experiment requests have no per-round work bound; private judge reasoning is written to @@ -379,10 +430,12 @@ packaging, entry points, and paths. no longer configures the judge, and verification requests cannot override its model, credentials, or endpoint. - For non-parity development, the judge can be overridden with - an exported `DISCOVERPHYSICS_JUDGE_MODEL` before startup. Official oracle and - parity runs must retain `claude-opus-4-6`. -- `parity_experiment.json` must be added after real experiments finish; registry - sizes, parity costs, and matching-agent metadata remain unset until then. + an exported `DISCOVERPHYSICS_JUDGE_MODEL` before startup. The reference oracle + and Opus parity configurations use `claude-opus-4-6`; the recorded historical + GLM reviewer runs used the GLM judge described above. +- `parity_experiment.json` records the two historical reviewer strata. Registry + publication and matching-agent acceptance remain unset; historical parity + spend has not been reconstructed. Run the adapter regression tests from the adapters repository root: diff --git a/src/discoverphysics/adapter_metadata.json b/src/discoverphysics/adapter_metadata.json index 828d152e..fb76f847 100644 --- a/src/discoverphysics/adapter_metadata.json +++ b/src/discoverphysics/adapter_metadata.json @@ -14,22 +14,46 @@ ], "adaptable": true, "notes": "Eleven supported n-body worlds at two seeds. The pinned registry has twelve worlds; wave requires the field engine and lacks batch benchmark normalization metadata. The pinned upstream config also names two worlds whose implementations are not released." + }, + { + "split": "public-worlds-four-seed-reviewer-variant", + "size": 44, + "harness": "agent", + "supported_agents": [ + "DiscoveryAgent@33b7fa9df96de9c35744efd181ca7e5a8dd60ad5" + ], + "adaptable": true, + "notes": "Historical four-seed reviewer universe: eleven n-body worlds at seeds 0–3. This is not the default 22-task dataset or a registry publication." } ], "harbor_adapter": [ { "split": "public-worlds", "adapted_benchmark_size": 22, - "parity_benchmark_size": 0, - "parity_sampling_rate": 0.0, + "parity_benchmark_size": 2, + "parity_sampling_rate": 0.09090909090909091, + "registry_benchmark_size": 0, + "added_agents": [ + "DiscoverPhysicsParityAgent@0.1.0" + ], + "parity_matching_agents": [], + "parity_unmatching_agents": [], + "parity_costs": "Unknown (historical spend not reconstructed)", + "notes": "Partial historical Gravity stream-v1 evidence: two tasks, three repetitions per side; run-score ranges overlap. This covers only Gravity in this runtime stratum; it does not establish full-adapter or migrated-head equivalence. Matching-agent acceptance remains unset. Oracle September 30: 22/22, zero errors/retries, mean explanation 0.95; dt=0.005 is the documented oracle exception. Parity HF PR: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270. Oracle HF PR: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/271. Reviewer reproduction patch/fork is deferred; standard CLI-agent validation, existing security/fidelity findings, and registry publication remain pending. Dataset PR: https://github.com/harbor-framework/harbor-datasets/pull/259. Adapter PR: https://github.com/harbor-framework/adapters/pull/9." + }, + { + "split": "public-worlds-four-seed-reviewer-variant", + "adapted_benchmark_size": 44, + "parity_benchmark_size": 6, + "parity_sampling_rate": 0.13636363636363635, "registry_benchmark_size": 0, "added_agents": [ "DiscoverPhysicsParityAgent@0.1.0" ], "parity_matching_agents": [], "parity_unmatching_agents": [], - "parity_costs": "Not run", - "notes": "Oracle discoverphysics-oracle-20260930-102507: 22/22 passed, zero errors/retries, mean explanation score 0.95. All task digests match the evaluated dataset; see validation/README.md. Oracle dt=0.005 is a documented exception to the prompt minimum. Pinned fractional grading-reference and Coulomb Easy final-fitting limitations are preserved; see README. Audrey Zheng coordinates parity; results, standard CLI-agent validation, HF evidence upload, and registry publication remain pending. Dataset PR: https://github.com/harbor-framework/harbor-datasets/pull/259. Adapter PR: https://github.com/harbor-framework/adapters/pull/9; original review: https://github.com/harbor-framework/harbor/pull/2974." + "parity_costs": "Unknown (historical spend not reconstructed)", + "notes": "Partial historical durable-v3 evidence: Yukawa seeds 2/3, Oscillator seeds 0/1, Hubble seeds 0/1; six complete tasks, three repetitions per side. Run-score ranges overlap. This four-seed reviewer variant is separate from Gravity stream-v1 and the default 22-task registry submission. Ether is excluded from the complete-case metric; strict first repetition is incomplete and original salvage remains separately qualified in the HF evidence. Matching-agent acceptance remains unset; current-head equivalence is not established. Reviewer reproduction patch/fork is deferred. Parity HF PR: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270." } ] } diff --git a/src/discoverphysics/parity_experiment.json b/src/discoverphysics/parity_experiment.json new file mode 100644 index 00000000..e144aa98 --- /dev/null +++ b/src/discoverphysics/parity_experiment.json @@ -0,0 +1,80 @@ +[ + { + "adapter_name": "discoverphysics", + "model": "zai-org/GLM-5.3-Flash", + "number_of_runs": 3, + "original_parity_repo": "", + "adapter_pr": [ + "https://github.com/harbor-framework/adapters/pull/9", + "https://github.com/harbor-framework/harbor/pull/2974" + ], + "dataset_pr": [ + "https://github.com/harbor-framework/harbor-datasets/pull/259" + ], + "parity_pr": [ + "https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270" + ], + "agent": "DiscoverPhysicsOpenAIParityAgent@sha256:eb6161f4eed35c0df80d7559bbcb4518550b33ff3fa6419ff63cb5ec1081280e", + "date": "2026-09-28", + "adapted_benchmark_size": 22, + "parity_benchmark_size": 2, + "notes": "Gravity stream-v1: gravity seeds 0/1, two tasks out of the 22-task two-seed universe. Date is the last recorded model-response UTC date. Do not pool with durable-v3. Partial historical reviewer evidence, not full-adapter acceptance or equivalence to migrated head 40bc08e6250211a618daa8db5ae12eca71f7ee51. Upstream 33b7fa9df96de9c35744efd181ca7e5a8dd60ad5; historical adapter baseline 059eef7a960c036760e7391cdf0da3faac59f9d9; runtime baseline 3c82380859d187957cfd5cd64802b076d9779550 plus reviewer changes. Discovery and judge both zai-org/GLM-5.3-Flash through W&B Inference on Modal; provider ID is not an immutable snapshot. Requested discovery/judge caps 131072/65536, 16 rounds, nominal fitting budget 180 seconds, openai SDK 1.102.0 with default 2 retries. Scores are fractions; uncertainty is sample SEM over three repetition scores. Coulomb and DeepSeek excluded retrospectively; selected configurations retain scored failures and empty/truncated responses. Public evidence contains raw outputs and provenance. original_parity_repo is empty because the reviewer reproduction patch/fork is deferred; the upstream repository is https://github.com/SampsonML/DiscoverPhysics. Full-precision noise and fitting/runtime changes remain qualifications; no new runs or regrading performed.", + "metrics": [ + { + "benchmark_name": "DiscoverPhysics/gravity stream-v1 (2 tasks)", + "metric": "law_success_rate (fraction; normalized_mse<0.1 and explanation_score>=0.75)", + "original": "1.00000000 ± 0.00000000", + "original_runs": [ + 1, + 1, + 1 + ], + "harbor": "1.00000000 ± 0.00000000", + "harbor_runs": [ + 1.0, + 1.0, + 1.0 + ] + } + ] + }, + { + "adapter_name": "discoverphysics", + "model": "zai-org/GLM-5.3-Flash", + "number_of_runs": 3, + "original_parity_repo": "", + "adapter_pr": [ + "https://github.com/harbor-framework/adapters/pull/9", + "https://github.com/harbor-framework/harbor/pull/2974" + ], + "dataset_pr": [ + "https://github.com/harbor-framework/harbor-datasets/pull/259" + ], + "parity_pr": [ + "https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270" + ], + "agent": "DiscoverPhysicsOpenAIParityAgent@sha256:6a5cc9f4861f7acef23df9611c58037b9a19eeaa94b7b06b330690ffa2d06734", + "date": "2026-10-09", + "adapted_benchmark_size": 44, + "parity_benchmark_size": 6, + "notes": "Persistent-worker streaming durable-v3: Yukawa seeds 2/3, Oscillator seeds 0/1, Hubble seeds 0/1, six complete tasks out of the 44-task four-seed universe. The four-seed universe is a historical reviewer variant, not the 22-task registry submission. Evidence dates October 7–9, 2026; date is the last recorded model-response UTC date. Each repetition equally weights these six tasks. Ether seeds 2/3 are excluded from this complete-case metric because Harbor repetition 1 seed 3 lacks a strict score. Its original salvaged zero remains a separately qualified sensitivity result, with an unresolved Harbor error-feedback discrepancy; supplemental outcomes are not substituted. Do not pool with Gravity. Partial historical reviewer evidence, not full-adapter acceptance or equivalence to migrated head 40bc08e6250211a618daa8db5ae12eca71f7ee51. Upstream 33b7fa9df96de9c35744efd181ca7e5a8dd60ad5; historical adapter baseline 059eef7a960c036760e7391cdf0da3faac59f9d9; runtime baseline 3c82380859d187957cfd5cd64802b076d9779550 plus reviewer changes. Discovery and judge both zai-org/GLM-5.3-Flash through W&B Inference on Modal; provider ID is not an immutable snapshot. Requested discovery/judge caps 131072/65536, 16 rounds, nominal fitting budget 180 seconds, openai SDK 1.102.0 with default 2 retries. Scores are fractions; uncertainty is sample SEM over three repetition scores. Coulomb and DeepSeek excluded retrospectively; selected configurations retain scored failures and empty/truncated responses. Public evidence contains raw outputs and provenance. original_parity_repo is empty because the reviewer reproduction patch/fork is deferred; the upstream repository is https://github.com/SampsonML/DiscoverPhysics. Full-precision noise and fitting/runtime changes remain qualifications; no new runs or regrading performed.", + "metrics": [ + { + "benchmark_name": "DiscoverPhysics/durable-v3 complete subset (6 tasks; excludes Ether)", + "metric": "law_success_rate (fraction; normalized_mse<0.1 and explanation_score>=0.75)", + "original": "0.11111111 ± 0.05555556", + "original_runs": [ + 0.16666666666666666, + 0.16666666666666666, + 0 + ], + "harbor": "0.16666667 ± 0.00000000", + "harbor_runs": [ + 0.16666666666666666, + 0.16666666666666666, + 0.16666666666666666 + ] + } + ] + } +] From af66f8ea68a39858086940132868572dbb11412f Mon Sep 17 00:00:00 2001 From: audreina17 Date: Thu, 22 Oct 2026 22:46:41 -0700 Subject: [PATCH 2/3] docs(discoverphysics): trim duplicated parity reporting --- src/discoverphysics/README.md | 36 +++++++--------------- src/discoverphysics/adapter_metadata.json | 4 +-- src/discoverphysics/parity_experiment.json | 4 +-- 3 files changed, 15 insertions(+), 29 deletions(-) diff --git a/src/discoverphysics/README.md b/src/discoverphysics/README.md index b80ed7d2..093f52d4 100644 --- a/src/discoverphysics/README.md +++ b/src/discoverphysics/README.md @@ -255,31 +255,17 @@ The six-task metric equally weights those tasks within each repetition; it does not pool Gravity or Ether. The four-seed universe has 44 possible tasks and is not the default 22-task dataset submitted for registration. -World-level rates, in the same fraction units, remain visible below and in the -[raw evidence and attempt ledger](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270): - -| World | Upstream runs | Harbor runs | Upstream mean ± SEM | Harbor mean ± SEM | -| --- | --- | --- | --- | --- | -| Gravity | 1, 1, 1 | 1, 1, 1 | 1.00000000 ± 0.00000000 | 1.00000000 ± 0.00000000 | -| Yukawa | 0, 0, 0 | 0, 0.5, 0 | 0.00000000 ± 0.00000000 | 0.16666667 ± 0.16666667 | -| Oscillator | 0, 0, 0 | 0, 0, 0 | 0.00000000 ± 0.00000000 | 0.00000000 ± 0.00000000 | -| Hubble | 0.5, 0.5, 0 | 0.5, 0, 0.5 | 0.33333333 ± 0.16666667 | 0.33333333 ± 0.16666667 | -| Ether, strict | 0, 0.5, 0.5 | missing, 1, 0.5 | 0.33333333 ± 0.16666667 | Incomplete | -| Ether, qualified original salvage | 0, 0.5, 0.5 | 0, 1, 0.5 | 0.33333333 ± 0.16666667 | 0.50000000 ± 0.28867513 | - -Ether's strict first Harbor repetition is incomplete. The original salvaged zero -has an unresolved Harbor error-feedback discrepancy, so it appears only in the -qualified sensitivity row. It is not included in the complete six-task metric. -Scored supplemental trials are not substituted for planned outcomes. -Oscillator's all-failure agreement provides limited discriminatory evidence. - -Coulomb and DeepSeek were excluded retrospectively. All outcomes in the selected -configurations remain visible, including scored failures and empty/truncated -responses. The inventory contains 67 attempts: 59 strict scored slots, six -unscored infrastructure attempts, and two scored supplemental trials outside the -60 planned slots. Discovery responses hit the length cap 83/425 times upstream -and 84/419 times in Harbor; these counts do not establish that all discrepancies -are random. +The [HF evidence](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270) +contains world-level scores, raw trajectories, verifier outputs, and the complete +attempt ledger. Ether has one missing strict score; its original salvaged zero +has an unresolved Harbor error-feedback discrepancy and is reported only as a +qualified sensitivity result. It is excluded from the six-task metric. +Oscillator's all-failure agreement has limited discriminatory power. + +Coulomb and DeepSeek were excluded retrospectively. Failures, empty/truncated +responses, and infrastructure attempts remain visible; scored supplemental +trials do not replace planned outcomes. Output-length limits alone do not +establish that discrepancies are random. Historical settings: W&B Inference on Modal, requested discovery/judge caps 131072/65536, 16 rounds, nominal fitting budget 180 seconds, and OpenAI SDK diff --git a/src/discoverphysics/adapter_metadata.json b/src/discoverphysics/adapter_metadata.json index fb76f847..354c5cf8 100644 --- a/src/discoverphysics/adapter_metadata.json +++ b/src/discoverphysics/adapter_metadata.json @@ -39,7 +39,7 @@ "parity_matching_agents": [], "parity_unmatching_agents": [], "parity_costs": "Unknown (historical spend not reconstructed)", - "notes": "Partial historical Gravity stream-v1 evidence: two tasks, three repetitions per side; run-score ranges overlap. This covers only Gravity in this runtime stratum; it does not establish full-adapter or migrated-head equivalence. Matching-agent acceptance remains unset. Oracle September 30: 22/22, zero errors/retries, mean explanation 0.95; dt=0.005 is the documented oracle exception. Parity HF PR: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270. Oracle HF PR: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/271. Reviewer reproduction patch/fork is deferred; standard CLI-agent validation, existing security/fidelity findings, and registry publication remain pending. Dataset PR: https://github.com/harbor-framework/harbor-datasets/pull/259. Adapter PR: https://github.com/harbor-framework/adapters/pull/9." + "notes": "Historical Gravity stream-v1 subset: 2/22 tasks, three repetitions per side; ranges overlap. Full-adapter and migrated-head acceptance remain unset. See parity_experiment.json and README for evidence links and qualifications. Oracle: 22/22; see validation/README.md and HF PR #271. CLI-agent validation and registry publication remain pending; reviewer patch/fork is deferred." }, { "split": "public-worlds-four-seed-reviewer-variant", @@ -53,7 +53,7 @@ "parity_matching_agents": [], "parity_unmatching_agents": [], "parity_costs": "Unknown (historical spend not reconstructed)", - "notes": "Partial historical durable-v3 evidence: Yukawa seeds 2/3, Oscillator seeds 0/1, Hubble seeds 0/1; six complete tasks, three repetitions per side. Run-score ranges overlap. This four-seed reviewer variant is separate from Gravity stream-v1 and the default 22-task registry submission. Ether is excluded from the complete-case metric; strict first repetition is incomplete and original salvage remains separately qualified in the HF evidence. Matching-agent acceptance remains unset; current-head equivalence is not established. Reviewer reproduction patch/fork is deferred. Parity HF PR: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270." + "notes": "Historical durable-v3 subset: Yukawa seeds 2/3, Oscillator and Hubble seeds 0/1; 6/44 tasks, three repetitions per side; ranges overlap. Separate from Gravity and the default registry dataset. Ether is excluded from the complete metric; see README and HF PR #270 for qualified salvage. Full-adapter and migrated-head acceptance remain unset; reviewer patch/fork is deferred." } ] } diff --git a/src/discoverphysics/parity_experiment.json b/src/discoverphysics/parity_experiment.json index e144aa98..852982de 100644 --- a/src/discoverphysics/parity_experiment.json +++ b/src/discoverphysics/parity_experiment.json @@ -18,7 +18,7 @@ "date": "2026-09-28", "adapted_benchmark_size": 22, "parity_benchmark_size": 2, - "notes": "Gravity stream-v1: gravity seeds 0/1, two tasks out of the 22-task two-seed universe. Date is the last recorded model-response UTC date. Do not pool with durable-v3. Partial historical reviewer evidence, not full-adapter acceptance or equivalence to migrated head 40bc08e6250211a618daa8db5ae12eca71f7ee51. Upstream 33b7fa9df96de9c35744efd181ca7e5a8dd60ad5; historical adapter baseline 059eef7a960c036760e7391cdf0da3faac59f9d9; runtime baseline 3c82380859d187957cfd5cd64802b076d9779550 plus reviewer changes. Discovery and judge both zai-org/GLM-5.3-Flash through W&B Inference on Modal; provider ID is not an immutable snapshot. Requested discovery/judge caps 131072/65536, 16 rounds, nominal fitting budget 180 seconds, openai SDK 1.102.0 with default 2 retries. Scores are fractions; uncertainty is sample SEM over three repetition scores. Coulomb and DeepSeek excluded retrospectively; selected configurations retain scored failures and empty/truncated responses. Public evidence contains raw outputs and provenance. original_parity_repo is empty because the reviewer reproduction patch/fork is deferred; the upstream repository is https://github.com/SampsonML/DiscoverPhysics. Full-precision noise and fitting/runtime changes remain qualifications; no new runs or regrading performed.", + "notes": "Gravity stream-v1, seeds 0/1; 2 of 22 tasks. Date is the last model-response UTC date. Do not pool with durable-v3. Partial historical evidence; full-adapter acceptance and migrated-head equivalence are not established. Both discovery and judging use the stated GLM model, a floating provider ID. Coulomb and DeepSeek were excluded retrospectively; failures and truncated replies are retained. Source pins, execution settings, and qualifications are in README and HF PR #270. original_parity_repo is empty because the reviewer patch/fork is deferred.", "metrics": [ { "benchmark_name": "DiscoverPhysics/gravity stream-v1 (2 tasks)", @@ -57,7 +57,7 @@ "date": "2026-10-09", "adapted_benchmark_size": 44, "parity_benchmark_size": 6, - "notes": "Persistent-worker streaming durable-v3: Yukawa seeds 2/3, Oscillator seeds 0/1, Hubble seeds 0/1, six complete tasks out of the 44-task four-seed universe. The four-seed universe is a historical reviewer variant, not the 22-task registry submission. Evidence dates October 7–9, 2026; date is the last recorded model-response UTC date. Each repetition equally weights these six tasks. Ether seeds 2/3 are excluded from this complete-case metric because Harbor repetition 1 seed 3 lacks a strict score. Its original salvaged zero remains a separately qualified sensitivity result, with an unresolved Harbor error-feedback discrepancy; supplemental outcomes are not substituted. Do not pool with Gravity. Partial historical reviewer evidence, not full-adapter acceptance or equivalence to migrated head 40bc08e6250211a618daa8db5ae12eca71f7ee51. Upstream 33b7fa9df96de9c35744efd181ca7e5a8dd60ad5; historical adapter baseline 059eef7a960c036760e7391cdf0da3faac59f9d9; runtime baseline 3c82380859d187957cfd5cd64802b076d9779550 plus reviewer changes. Discovery and judge both zai-org/GLM-5.3-Flash through W&B Inference on Modal; provider ID is not an immutable snapshot. Requested discovery/judge caps 131072/65536, 16 rounds, nominal fitting budget 180 seconds, openai SDK 1.102.0 with default 2 retries. Scores are fractions; uncertainty is sample SEM over three repetition scores. Coulomb and DeepSeek excluded retrospectively; selected configurations retain scored failures and empty/truncated responses. Public evidence contains raw outputs and provenance. original_parity_repo is empty because the reviewer reproduction patch/fork is deferred; the upstream repository is https://github.com/SampsonML/DiscoverPhysics. Full-precision noise and fitting/runtime changes remain qualifications; no new runs or regrading performed.", + "notes": "Durable-v3 persistent worker: Yukawa seeds 2/3, Oscillator and Hubble seeds 0/1; 6 of 44 tasks in the historical four-seed universe. Each repetition weights the six tasks equally. Evidence dates October 7–9, 2026; date is the last model-response UTC date. Ether has one missing strict score; its qualified salvage remains in HF PR #270 and is excluded from this metric. Do not pool with Gravity. Partial historical evidence; full-adapter acceptance and migrated-head equivalence are not established. Both discovery and judging use the stated GLM model, a floating provider ID. Coulomb and DeepSeek were excluded retrospectively; failures and truncated replies are retained. Source pins, execution settings, and qualifications are in README and HF PR #270. original_parity_repo is empty because the reviewer patch/fork is deferred.", "metrics": [ { "benchmark_name": "DiscoverPhysics/durable-v3 complete subset (6 tasks; excludes Ether)", From 91251d953a14b0c6c87c429d715fc9acea2c5e4e Mon Sep 17 00:00:00 2001 From: audreina17 Date: Thu, 22 Oct 2026 22:52:33 -0700 Subject: [PATCH 3/3] docs(discoverphysics): focus parity summary on tested subsets --- src/discoverphysics/README.md | 25 +++++++++------------- src/discoverphysics/adapter_metadata.json | 4 ++-- src/discoverphysics/parity_experiment.json | 6 +++--- 3 files changed, 15 insertions(+), 20 deletions(-) diff --git a/src/discoverphysics/README.md b/src/discoverphysics/README.md index 093f52d4..0768816c 100644 --- a/src/discoverphysics/README.md +++ b/src/discoverphysics/README.md @@ -238,8 +238,8 @@ uv run discoverphysics \ The [partial historical parity evidence](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270) records three repetitions per side using `zai-org/GLM-5.3-Flash` for both discovery and judging. Scores below -are fractions reported as mean ± sample SEM. The complete comparisons have -overlapping raw run-score ranges. These results do not establish full-adapter +are fractions reported as mean ± sample SEM. The range-overlap parity criterion +is met for both tested subsets below. These results do not establish full-adapter acceptance or equivalence to the migrated adapter head. | Agent | Model | Metric | Number of Runs | Dataset Size | Original Benchmark Performance | Harbor Adapter Performance | @@ -251,21 +251,16 @@ acceptance or equivalence to the migrated adapter head. separately, including exact agent SHA-256 versions and raw repetition arrays. Gravity uses stream-v1 at seeds 0/1. Durable-v3 uses a persistent worker and a four-seed universe: Yukawa seeds 2/3, Oscillator seeds 0/1, and Hubble seeds 0/1. -The six-task metric equally weights those tasks within each repetition; it does -not pool Gravity or Ether. The four-seed universe has 44 possible tasks and is -not the default 22-task dataset submitted for registration. +The six-task metric equally weights Yukawa, Oscillator, and Hubble within each +repetition and stays separate from Gravity. The four-seed universe has 44 +possible tasks and is not the default 22-task dataset submitted for registration. The [HF evidence](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270) -contains world-level scores, raw trajectories, verifier outputs, and the complete -attempt ledger. Ether has one missing strict score; its original salvaged zero -has an unresolved Harbor error-feedback discrepancy and is reported only as a -qualified sensitivity result. It is excluded from the six-task metric. -Oscillator's all-failure agreement has limited discriminatory power. - -Coulomb and DeepSeek were excluded retrospectively. Failures, empty/truncated -responses, and infrastructure attempts remain visible; scored supplemental -trials do not replace planned outcomes. Output-length limits alone do not -establish that discrepancies are random. +contains world-level scores, raw trajectories, verifier outputs, provenance, +and the complete attempt ledger. Failures and empty/truncated responses in the +tested subsets remain in the reported scores. Oscillator's all-failure agreement +has limited discriminatory power. Output-length limits alone do not establish +that discrepancies are random. Historical settings: W&B Inference on Modal, requested discovery/judge caps 131072/65536, 16 rounds, nominal fitting budget 180 seconds, and OpenAI SDK diff --git a/src/discoverphysics/adapter_metadata.json b/src/discoverphysics/adapter_metadata.json index 354c5cf8..446d1b66 100644 --- a/src/discoverphysics/adapter_metadata.json +++ b/src/discoverphysics/adapter_metadata.json @@ -39,7 +39,7 @@ "parity_matching_agents": [], "parity_unmatching_agents": [], "parity_costs": "Unknown (historical spend not reconstructed)", - "notes": "Historical Gravity stream-v1 subset: 2/22 tasks, three repetitions per side; ranges overlap. Full-adapter and migrated-head acceptance remain unset. See parity_experiment.json and README for evidence links and qualifications. Oracle: 22/22; see validation/README.md and HF PR #271. CLI-agent validation and registry publication remain pending; reviewer patch/fork is deferred." + "notes": "Historical Gravity stream-v1 subset: 2/22 tasks, three repetitions per side; the range-overlap parity criterion is met for this subset. Full-adapter and migrated-head acceptance remain unset. See parity_experiment.json and README for evidence links and qualifications. Oracle: 22/22; see validation/README.md and HF PR #271. CLI-agent validation and registry publication remain pending; reviewer patch/fork is deferred." }, { "split": "public-worlds-four-seed-reviewer-variant", @@ -53,7 +53,7 @@ "parity_matching_agents": [], "parity_unmatching_agents": [], "parity_costs": "Unknown (historical spend not reconstructed)", - "notes": "Historical durable-v3 subset: Yukawa seeds 2/3, Oscillator and Hubble seeds 0/1; 6/44 tasks, three repetitions per side; ranges overlap. Separate from Gravity and the default registry dataset. Ether is excluded from the complete metric; see README and HF PR #270 for qualified salvage. Full-adapter and migrated-head acceptance remain unset; reviewer patch/fork is deferred." + "notes": "Historical durable-v3 subset: Yukawa seeds 2/3, Oscillator and Hubble seeds 0/1; 6/44 tasks, three repetitions per side; the range-overlap parity criterion is met for this subset. Separate from Gravity and the default registry dataset. Full-adapter and migrated-head acceptance remain unset; reviewer patch/fork is deferred." } ] } diff --git a/src/discoverphysics/parity_experiment.json b/src/discoverphysics/parity_experiment.json index 852982de..67ad8155 100644 --- a/src/discoverphysics/parity_experiment.json +++ b/src/discoverphysics/parity_experiment.json @@ -18,7 +18,7 @@ "date": "2026-09-28", "adapted_benchmark_size": 22, "parity_benchmark_size": 2, - "notes": "Gravity stream-v1, seeds 0/1; 2 of 22 tasks. Date is the last model-response UTC date. Do not pool with durable-v3. Partial historical evidence; full-adapter acceptance and migrated-head equivalence are not established. Both discovery and judging use the stated GLM model, a floating provider ID. Coulomb and DeepSeek were excluded retrospectively; failures and truncated replies are retained. Source pins, execution settings, and qualifications are in README and HF PR #270. original_parity_repo is empty because the reviewer patch/fork is deferred.", + "notes": "The run-score range-overlap parity criterion is met for this tested subset. Gravity stream-v1, seeds 0/1; 2 of 22 tasks. Date is the last model-response UTC date. Do not pool with durable-v3. Partial historical evidence; full-adapter acceptance and migrated-head equivalence are not established. Both discovery and judging use the stated GLM model, a floating provider ID. Failures and truncated replies in these tested subsets are retained. Source pins, execution settings, and qualifications are in README and HF PR #270. original_parity_repo is empty because the reviewer patch/fork is deferred.", "metrics": [ { "benchmark_name": "DiscoverPhysics/gravity stream-v1 (2 tasks)", @@ -57,10 +57,10 @@ "date": "2026-10-09", "adapted_benchmark_size": 44, "parity_benchmark_size": 6, - "notes": "Durable-v3 persistent worker: Yukawa seeds 2/3, Oscillator and Hubble seeds 0/1; 6 of 44 tasks in the historical four-seed universe. Each repetition weights the six tasks equally. Evidence dates October 7–9, 2026; date is the last model-response UTC date. Ether has one missing strict score; its qualified salvage remains in HF PR #270 and is excluded from this metric. Do not pool with Gravity. Partial historical evidence; full-adapter acceptance and migrated-head equivalence are not established. Both discovery and judging use the stated GLM model, a floating provider ID. Coulomb and DeepSeek were excluded retrospectively; failures and truncated replies are retained. Source pins, execution settings, and qualifications are in README and HF PR #270. original_parity_repo is empty because the reviewer patch/fork is deferred.", + "notes": "The run-score range-overlap parity criterion is met for this tested subset. Durable-v3 persistent worker: Yukawa seeds 2/3, Oscillator and Hubble seeds 0/1; 6 of 44 tasks in the historical four-seed universe. Each repetition weights the six tasks equally. Evidence dates October 7–9, 2026; date is the last model-response UTC date. Do not pool with Gravity. Partial historical evidence; full-adapter acceptance and migrated-head equivalence are not established. Both discovery and judging use the stated GLM model, a floating provider ID. Failures and truncated replies in these tested subsets are retained. Source pins, execution settings, and qualifications are in README and HF PR #270. original_parity_repo is empty because the reviewer patch/fork is deferred.", "metrics": [ { - "benchmark_name": "DiscoverPhysics/durable-v3 complete subset (6 tasks; excludes Ether)", + "benchmark_name": "DiscoverPhysics/durable-v3 Yukawa, Oscillator, Hubble (6 tasks)", "metric": "law_success_rate (fraction; normalized_mse<0.1 and explanation_score>=0.75)", "original": "0.11111111 ± 0.05555556", "original_runs": [