Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 65 additions & 31 deletions src/discoverphysics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,28 +236,59 @@ uv run discoverphysics \

### Parity type: Scenario 3 (custom upstream agent)

Parity is being run by Audrey Zheng, as coordinated in the
[migration request](https://github.com/harbor-framework/harbor/pull/2974#issuecomment-6004953490);
reviewed results are not yet recorded here. Migration does not launch duplicate runs.
The experiment uses the full 22-task
public suite, `DiscoveryAgent@33b7fa9d` on upstream and
`DiscoverPhysicsParityAgent@0.1.0` in Harbor, `claude-opus-4-7` as agent,
`claude-opus-4-6` as judge, critic disabled, N-body engine, noise fraction
`0.05`, seeds 0 and 1, 16 rounds, and MSE fitting enabled. Both sides use
Python 3.12 and `anthropic==0.64.0` for model calls.

The parity plan must be approved by the Harbor team before API spend. Execution
then proceeds symmetrically in the required order:

1. Run the fixed six-task sanity set on both sides.
2. Inspect errors and trajectories, then run one full 22-task repetition on
both sides.
3. Only if the full-run score ranges are compatible, run repetitions two and
three on both sides with the same commits and configuration.
The [partial historical parity evidence](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270) records three repetitions
per side using `zai-org/GLM-5.3-Flash` for both discovery and judging. Scores below
are fractions reported as mean ± sample SEM. The range-overlap parity criterion
is met for both tested subsets below. These results do not establish full-adapter
acceptance or equivalence to the migrated adapter head.

| Agent | Model | Metric | Number of Runs | Dataset Size | Original Benchmark Performance | Harbor Adapter Performance |
| --- | --- | --- | ---: | ---: | --- | --- |
| Not run | Not run | Trial pass rate | 0 | 0 | Pending | Pending |
| --- | --- | --- | ---: | --- | --- | --- |
| Reviewer parity agent, stream-v1 | zai-org/GLM-5.3-Flash | Gravity success rate | 3 | 2/22 (9.09%) | 1.00000000 ± 0.00000000 | 1.00000000 ± 0.00000000 |
| Reviewer parity agent, durable-v3 | zai-org/GLM-5.3-Flash | Complete six-task success rate | 3 | 6/44 (13.64%) | 0.11111111 ± 0.05555556 | 0.16666667 ± 0.00000000 |

[parity_experiment.json](parity_experiment.json) records both configuration strata
separately, including exact agent SHA-256 versions and raw repetition arrays.
Gravity uses stream-v1 at seeds 0/1. Durable-v3 uses a persistent worker and a
four-seed universe: Yukawa seeds 2/3, Oscillator seeds 0/1, and Hubble seeds 0/1.
The six-task metric equally weights Yukawa, Oscillator, and Hubble within each
repetition and stays separate from Gravity. The four-seed universe has 44
possible tasks and is not the default 22-task dataset submitted for registration.

The [HF evidence](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270)
contains world-level scores, raw trajectories, verifier outputs, provenance,
and the complete attempt ledger. Failures and empty/truncated responses in the
tested subsets remain in the reported scores. Oscillator's all-failure agreement
has limited discriminatory power. Output-length limits alone do not establish
that discrepancies are random.

Historical settings: W&B Inference on Modal, requested discovery/judge caps
131072/65536, 16 rounds, nominal fitting budget 180 seconds, and OpenAI SDK
1.102.0. The model ID is not an immutable snapshot. Upstream is pinned to
`33b7fa9df96de9c35744efd181ca7e5a8dd60ad5`; the historical adapter baseline is
`059eef7a960c036760e7391cdf0da3faac59f9d9` and runtime baseline is
`3c82380859d187957cfd5cd64802b076d9779550`, with reviewer changes. Full-precision
noise and fitting/runtime differences remain qualifications. No new run or
regrading was performed for this integration.

The reviewer-runtime patch/fork is deferred. `original_parity_repo` therefore
remains empty rather than linking a repository that cannot reproduce these
runs. The HF upload retains the recorded configurations and provenance; the
commands below describe the separate Opus reference plan and do not reproduce
the historical GLM evidence.

### Reference Opus parity plan (not the recorded GLM results)

The existing reference configuration targets the full 22-task public suite,
`DiscoveryAgent@33b7fa9d` upstream and `DiscoverPhysicsParityAgent@0.1.0` in
Harbor, `claude-opus-4-7` for discovery and `claude-opus-4-6` for judging, critic
disabled, N-body engine, noise fraction `0.05`, seeds 0/1, 16 rounds, MSE fitting,
Python 3.12, and `anthropic==0.64.0`.

Before new API spend, the Harbor team must approve the plan: run the fixed
six-task sanity set on both sides, inspect errors and trajectories, run the
first full repetition, then run repetitions two and three only if the first
comparison is compatible. This metadata integration launches no new runs.

Original-side reproduction from the pinned DiscoverPhysics checkout:

Expand Down Expand Up @@ -303,7 +334,7 @@ uv run --project src/discoverphysics --extra harbor harbor run -c src/discoverph

Report trial pass rate as mean ± sample SEM and retain every raw per-run score.
Parity passes only if the original and Harbor raw run-score ranges overlap. Do
not populate `parity_experiment.json` until real results exist. A separate
not treat historical subset overlap as full-suite validation. A separate
standard Harbor CLI-agent run is also required as Harbor-only compatibility
evidence; it is not an original-versus-adapter parity claim.

Expand All @@ -318,12 +349,13 @@ packaging, entry points, and paths.
[evaluated snapshot](https://github.com/Randl/harbor-datasets/tree/4952f1fe2e0104d6ae5467c52444dbd908b13828/datasets/discoverphysics).
Registry publication is pending. For repackaging, follow the repository's
[dataset registration guide](../../docs/adapters.mdx#step-8-register-the-dataset).
- [Latest oracle evidence](validation/README.md): 22/22 passed. Full logs and
experiment artifacts are prepared for `harborframework/parity-experiments`;
the public upload link is pending. Use the repository's
[upload skill](../../skills/upload-parity-experiments/SKILL.md) for publication.
- Reviewer-coordinated parity and the separate standard CLI-agent check remain
pending. An oracle pass does not establish either result.
- [Latest oracle evidence](validation/README.md): 22/22 passed. Full sanitized
logs, configurations, experiments, and exact tasks are published in
[HF oracle PR #271](https://huggingface.co/datasets/harborframework/parity-experiments/discussions/271).
- Partial historical parity is recorded above and in `parity_experiment.json`.
Full-adapter acceptance, current-head equivalence, and the separate standard
CLI-agent check remain pending. The reviewer-runtime patch/fork is deferred.
An oracle pass does not establish parity or CLI-agent compatibility.
- Open review limitations: the root-only verifier token shares the main container
with the agent; discovery credentials expose native parity prompts; experiment
requests have no per-round work bound; private judge reasoning is written to
Expand Down Expand Up @@ -379,10 +411,12 @@ packaging, entry points, and paths.
no longer configures the judge, and verification requests cannot override its
model, credentials, or endpoint.
- For non-parity development, the judge can be overridden with
an exported `DISCOVERPHYSICS_JUDGE_MODEL` before startup. Official oracle and
parity runs must retain `claude-opus-4-6`.
- `parity_experiment.json` must be added after real experiments finish; registry
sizes, parity costs, and matching-agent metadata remain unset until then.
an exported `DISCOVERPHYSICS_JUDGE_MODEL` before startup. The reference oracle
and Opus parity configurations use `claude-opus-4-6`; the recorded historical
GLM reviewer runs used the GLM judge described above.
- `parity_experiment.json` records the two historical reviewer strata. Registry
publication and matching-agent acceptance remain unset; historical parity
spend has not been reconstructed.

Run the adapter regression tests from the adapters repository root:

Expand Down
32 changes: 28 additions & 4 deletions src/discoverphysics/adapter_metadata.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,22 +14,46 @@
],
"adaptable": true,
"notes": "Eleven supported n-body worlds at two seeds. The pinned registry has twelve worlds; wave requires the field engine and lacks batch benchmark normalization metadata. The pinned upstream config also names two worlds whose implementations are not released."
},
{
"split": "public-worlds-four-seed-reviewer-variant",
"size": 44,
"harness": "agent",
"supported_agents": [
"DiscoveryAgent@33b7fa9df96de9c35744efd181ca7e5a8dd60ad5"
],
"adaptable": true,
"notes": "Historical four-seed reviewer universe: eleven n-body worlds at seeds 0–3. This is not the default 22-task dataset or a registry publication."
}
],
"harbor_adapter": [
{
"split": "public-worlds",
"adapted_benchmark_size": 22,
"parity_benchmark_size": 0,
"parity_sampling_rate": 0.0,
"parity_benchmark_size": 2,
"parity_sampling_rate": 0.09090909090909091,
"registry_benchmark_size": 0,
"added_agents": [
"DiscoverPhysicsParityAgent@0.1.0"
],
"parity_matching_agents": [],
"parity_unmatching_agents": [],
"parity_costs": "Unknown (historical spend not reconstructed)",
"notes": "Historical Gravity stream-v1 subset: 2/22 tasks, three repetitions per side; the range-overlap parity criterion is met for this subset. Full-adapter and migrated-head acceptance remain unset. See parity_experiment.json and README for evidence links and qualifications. Oracle: 22/22; see validation/README.md and HF PR #271. CLI-agent validation and registry publication remain pending; reviewer patch/fork is deferred."
},
{
"split": "public-worlds-four-seed-reviewer-variant",
"adapted_benchmark_size": 44,
"parity_benchmark_size": 6,
"parity_sampling_rate": 0.13636363636363635,
"registry_benchmark_size": 0,
"added_agents": [
"DiscoverPhysicsParityAgent@0.1.0"
],
"parity_matching_agents": [],
"parity_unmatching_agents": [],
"parity_costs": "Not run",
"notes": "Oracle discoverphysics-oracle-20260930-102507: 22/22 passed, zero errors/retries, mean explanation score 0.95. All task digests match the evaluated dataset; see validation/README.md. Oracle dt=0.005 is a documented exception to the prompt minimum. Pinned fractional grading-reference and Coulomb Easy final-fitting limitations are preserved; see README. Audrey Zheng coordinates parity; results, standard CLI-agent validation, HF evidence upload, and registry publication remain pending. Dataset PR: https://github.com/harbor-framework/harbor-datasets/pull/259. Adapter PR: https://github.com/harbor-framework/adapters/pull/9; original review: https://github.com/harbor-framework/harbor/pull/2974."
"parity_costs": "Unknown (historical spend not reconstructed)",
"notes": "Historical durable-v3 subset: Yukawa seeds 2/3, Oscillator and Hubble seeds 0/1; 6/44 tasks, three repetitions per side; the range-overlap parity criterion is met for this subset. Separate from Gravity and the default registry dataset. Full-adapter and migrated-head acceptance remain unset; reviewer patch/fork is deferred."
}
]
}
Expand Down
80 changes: 80 additions & 0 deletions src/discoverphysics/parity_experiment.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
[
{
"adapter_name": "discoverphysics",
"model": "zai-org/GLM-5.3-Flash",
"number_of_runs": 3,
"original_parity_repo": "",
"adapter_pr": [
"https://github.com/harbor-framework/adapters/pull/9",
"https://github.com/harbor-framework/harbor/pull/2974"
],
"dataset_pr": [
"https://github.com/harbor-framework/harbor-datasets/pull/259"
],
"parity_pr": [
"https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270"
],
"agent": "DiscoverPhysicsOpenAIParityAgent@sha256:eb6161f4eed35c0df80d7559bbcb4518550b33ff3fa6419ff63cb5ec1081280e",
"date": "2026-09-28",
"adapted_benchmark_size": 22,
"parity_benchmark_size": 2,
"notes": "The run-score range-overlap parity criterion is met for this tested subset. Gravity stream-v1, seeds 0/1; 2 of 22 tasks. Date is the last model-response UTC date. Do not pool with durable-v3. Partial historical evidence; full-adapter acceptance and migrated-head equivalence are not established. Both discovery and judging use the stated GLM model, a floating provider ID. Failures and truncated replies in these tested subsets are retained. Source pins, execution settings, and qualifications are in README and HF PR #270. original_parity_repo is empty because the reviewer patch/fork is deferred.",
"metrics": [
{
"benchmark_name": "DiscoverPhysics/gravity stream-v1 (2 tasks)",
"metric": "law_success_rate (fraction; normalized_mse<0.1 and explanation_score>=0.75)",
"original": "1.00000000 ± 0.00000000",
"original_runs": [
1,
1,
1
],
"harbor": "1.00000000 ± 0.00000000",
"harbor_runs": [
1.0,
1.0,
1.0
]
}
]
},
{
"adapter_name": "discoverphysics",
"model": "zai-org/GLM-5.3-Flash",
"number_of_runs": 3,
"original_parity_repo": "",
"adapter_pr": [
"https://github.com/harbor-framework/adapters/pull/9",
"https://github.com/harbor-framework/harbor/pull/2974"
],
"dataset_pr": [
"https://github.com/harbor-framework/harbor-datasets/pull/259"
],
"parity_pr": [
"https://huggingface.co/datasets/harborframework/parity-experiments/discussions/270"
],
"agent": "DiscoverPhysicsOpenAIParityAgent@sha256:6a5cc9f4861f7acef23df9611c58037b9a19eeaa94b7b06b330690ffa2d06734",
"date": "2026-10-09",
"adapted_benchmark_size": 44,
"parity_benchmark_size": 6,
"notes": "The run-score range-overlap parity criterion is met for this tested subset. Durable-v3 persistent worker: Yukawa seeds 2/3, Oscillator and Hubble seeds 0/1; 6 of 44 tasks in the historical four-seed universe. Each repetition weights the six tasks equally. Evidence dates October 7–9, 2026; date is the last model-response UTC date. Do not pool with Gravity. Partial historical evidence; full-adapter acceptance and migrated-head equivalence are not established. Both discovery and judging use the stated GLM model, a floating provider ID. Failures and truncated replies in these tested subsets are retained. Source pins, execution settings, and qualifications are in README and HF PR #270. original_parity_repo is empty because the reviewer patch/fork is deferred.",
"metrics": [
{
"benchmark_name": "DiscoverPhysics/durable-v3 Yukawa, Oscillator, Hubble (6 tasks)",
"metric": "law_success_rate (fraction; normalized_mse<0.1 and explanation_score>=0.75)",
"original": "0.11111111 ± 0.05555556",
"original_runs": [
0.16666666666666666,
0.16666666666666666,
0
],
"harbor": "0.16666667 ± 0.00000000",
"harbor_runs": [
0.16666666666666666,
0.16666666666666666,
0.16666666666666666
]
}
]
}
]