data: add independent DeepSeek V4 Pro anchoring replication - #5
Open
NineThoughts0521 wants to merge 2 commits into
Open
data: add independent DeepSeek V4 Pro anchoring replication#5NineThoughts0521 wants to merge 2 commits into
NineThoughts0521 wants to merge 2 commits into
Conversation
- 增加 Minimal-Full 消融 preset 与零成本 schema gate - 发布 3+1 结果、成本、轨迹聚合和证据哈希 - 保留 OpenCode partial/replacement 与第三方统计边界
- 区分维护者正式统计与 NineThoughts0521 独立结果 - 接入 Minimal-Full、trajectory、成本与证据限制 - 如实披露 OpenCode partial 和 replacement
There was a problem hiding this comment.
Pull request overview
This PR adds a third-party, independent replication package for DeepSeek V4 Pro “first-request tool-schema anchoring” on the frozen Project2 V4.1b baseline, including a new minimal-full ablation preset plus scripts and published (privacy-preserving) derived artifacts.
Changes:
- Add
minimal-fullDSH preset (full Standard tool catalog from request 1, Minimal complete system condition) and document its role in the ablation. - Add an
experiments/deepseek-v4-pro-anchoring/bundle (runners, schema gate, analyzers, comparison builder, and derived/public artifacts). - Update top-level docs/scoreboard indices to reference the independent replication, and extend
.gitignoreto keep private evidence out of Git.
Reviewed changes
Copilot reviewed 55 out of 60 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/deepseek-harness-presets/README.md | Documents the new minimal-full preset and its validation requirements. |
| tools/deepseek-harness-presets/minimal-full/preset.yml | Registers the minimal-full preset (name/description/order). |
| tools/deepseek-harness-presets/minimal-full/agent.cordis.yml | Defines the minimal-full agent-plane composition (Minimal system condition + Standard tool roster). |
| README.md | Adds a section summarizing the independent replication and links to the report and experiment directory. |
| experiments/deepseek-v4-pro-anchoring/scripts/validate_presets.py | Validates schema-gate request snapshots via hashes and tool catalogs. |
| experiments/deepseek-v4-pro-anchoring/scripts/run_project2.ps1 | Automates DSH runs + evaluator execution + session analysis for Project2. |
| experiments/deepseek-v4-pro-anchoring/scripts/run_opencode_replacement_detached.cmd | Runs the OpenCode replacement runner in a detached wrapper style with redirected logs. |
| experiments/deepseek-v4-pro-anchoring/scripts/run_opencode_agent.py | Runs OpenCode against DeepSeek API with balance polling and exports a private session artifact. |
| experiments/deepseek-v4-pro-anchoring/scripts/run_mock_gate.ps1 | Executes a zero-cost mock schema gate for Standard/Minimal-Full/Anchored tool-catalog assertions. |
| experiments/deepseek-v4-pro-anchoring/scripts/mock_deepseek_server.mjs | Local mock DeepSeek-compatible SSE server used for the schema gate. |
| experiments/deepseek-v4-pro-anchoring/scripts/launch_detached.py | Helper to launch a Python runner with stdout/stderr redirected. |
| experiments/deepseek-v4-pro-anchoring/scripts/extract_candidate_prompt.py | Extracts the candidate prompt block from CANDIDATE_PROMPT.md and emits its SHA-256. |
| experiments/deepseek-v4-pro-anchoring/scripts/dsh_session_driver.mjs | Drives DSH Web via RPC to run a prompt and export session history/meta. |
| experiments/deepseek-v4-pro-anchoring/scripts/collect_price.py | Captures a DeepSeek pricing-page snapshot and writes a structured price record. |
| experiments/deepseek-v4-pro-anchoring/scripts/collect_balance.py | Captures a DeepSeek balance snapshot and writes a structured balance record. |
| experiments/deepseek-v4-pro-anchoring/scripts/capture_baseline.py | Captures environment + pinned versions + preset tree hashes for reproducibility. |
| experiments/deepseek-v4-pro-anchoring/scripts/calculate_balance_delta.py | Computes per-window balance deltas between two balance snapshots. |
| experiments/deepseek-v4-pro-anchoring/scripts/build_evidence_manifest.py | Builds a manifest of public artifact hashes and private-raw hash pointers. |
| experiments/deepseek-v4-pro-anchoring/scripts/build_comparison.py | Aggregates per-run artifacts into a machine-readable comparison JSON. |
| experiments/deepseek-v4-pro-anchoring/scripts/analyze_session.py | Derives redacted/public stats from a DSH session JSONL and evaluator summary. |
| experiments/deepseek-v4-pro-anchoring/scripts/analyze_opencode_session.py | Derives redacted/public stats from an OpenCode export + gate/price/balance inputs. |
| experiments/deepseek-v4-pro-anchoring/run-matrix.json | Declares the run plan/status matrix for the 3+1 bundle. |
| experiments/deepseek-v4-pro-anchoring/RESULTS.md | Publishes the summarized 3+1 results table and bounded conclusions. |
| experiments/deepseek-v4-pro-anchoring/README.md | Provides experiment directory overview and privacy/evidence boundaries. |
| experiments/deepseek-v4-pro-anchoring/preregistration.json | Captures preregistration metadata, pins, budget, and amendments. |
| experiments/deepseek-v4-pro-anchoring/mock-prompt.txt | Prompt used for the mock schema gate. |
| experiments/deepseek-v4-pro-anchoring/artifacts/schema-gate.json | Published output of the mock schema gate assertions. |
| experiments/deepseek-v4-pro-anchoring/artifacts/runs/P2-20260815-04b-opencode-replacement.json | Published derived artifact for the OpenCode replacement run. |
| experiments/deepseek-v4-pro-anchoring/artifacts/runs/P2-20260815-04b-opencode-replacement-balance.json | Published balance delta for the OpenCode replacement run. |
| experiments/deepseek-v4-pro-anchoring/artifacts/runs/P2-20260815-04-opencode-partial.json | Published derived artifact for the preserved partial OpenCode run. |
| experiments/deepseek-v4-pro-anchoring/artifacts/runs/P2-20260815-04-opencode-partial-balance.json | Published balance delta for the preserved partial OpenCode run. |
| experiments/deepseek-v4-pro-anchoring/artifacts/runs/P2-20260815-03-minimal-full-balance.json | Published balance delta for the Minimal-Full run. |
| experiments/deepseek-v4-pro-anchoring/artifacts/runs/P2-20260815-02-standard-balance.json | Published balance delta for the Standard run. |
| experiments/deepseek-v4-pro-anchoring/artifacts/runs/P2-20260815-01-anchored-balance.json | Published balance delta for the Anchored run. |
| experiments/deepseek-v4-pro-anchoring/artifacts/price-snapshot.json | Published pricing snapshot used for cost recomputation. |
| experiments/deepseek-v4-pro-anchoring/artifacts/price-P2-20260815-04b-opencode-replacement.json | Per-run pricing snapshot for OpenCode replacement. |
| experiments/deepseek-v4-pro-anchoring/artifacts/price-P2-20260815-04-opencode.json | Per-run pricing snapshot for initial OpenCode attempt. |
| experiments/deepseek-v4-pro-anchoring/artifacts/price-P2-20260815-03-minimal-full.json | Per-run pricing snapshot for Minimal-Full. |
| experiments/deepseek-v4-pro-anchoring/artifacts/price-P2-20260815-02-standard.json | Per-run pricing snapshot for Standard. |
| experiments/deepseek-v4-pro-anchoring/artifacts/price-P2-20260815-01-anchored.json | Per-run pricing snapshot for Anchored. |
| experiments/deepseek-v4-pro-anchoring/artifacts/opencode-gate.json | Published gate confirming OpenCode configuration/tool catalog assumptions. |
| experiments/deepseek-v4-pro-anchoring/artifacts/metadata-corrections.json | Records post-run corrections to derived metadata (with reasons). |
| experiments/deepseek-v4-pro-anchoring/artifacts/infrastructure-events.json | Records infra events and retry boundaries for the run set. |
| experiments/deepseek-v4-pro-anchoring/artifacts/evidence-manifest.json | Published manifest of public artifact hashes and private evidence hashes. |
| experiments/deepseek-v4-pro-anchoring/artifacts/environment-baseline.json | Published environment/pin snapshot including preset tree hashes. |
| experiments/deepseek-v4-pro-anchoring/artifacts/deepswe-gate.json | Published environment gate output for deferred DeepSWE pairing. |
| experiments/deepseek-v4-pro-anchoring/artifacts/balance-events.json | Published balance window events and per-run window hashes/deltas. |
| evaluator/trajectory_evidence/README.md | Clarifies separation between maintainer baseline evidence and third-party replication aggregates. |
| evaluator/reports/v4.1b_scoreboard.md | Adds a “third-party replication” section excluded from formal n. |
| evaluator/reports/README.md | Adds the independent replication report link to the report index. |
| docs/v4.1/README.md | Adds the independent replication report link to the v4.1 doc index. |
| docs/v4.1/DEEPSEEK_V4_TRIGGER_MECHANISM_EXPERIMENTS_20260814.md | Adds an update section referencing the third-party ablation/replication. |
| docs/v4.1/DEEPSEEK_V4_TRAJECTORY_ANALYSIS_20260814.md | Adds fingerprint summary rows for the third-party runs. |
| docs/v4.1/DEEPSEEK_V4_PRO_INDEPENDENT_REPLICATION_20260815.md | New full report describing methodology, boundaries, results, and limitations. |
| docs/v4.1/DEEPSEEK_V4_PRO_HARNESS_ANALYSIS_20260814.md | Adds an appendix referencing the third-party independent replication findings. |
| .gitignore | Ignores private experiment evidence directories and environment files. |
Suppressed comments (1)
experiments/deepseek-v4-pro-anchoring/scripts/validate_presets.py:28
load()iteratesvaluebefore confirming the JSON root is a list, so a non-list input will raiseTypeErrorinstead of the intended validation error. Validate the type first, and (optionally) filter only dict entries before accessing.get().
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+13
to
+20
| def request_fingerprint(request: dict) -> dict: | ||
| tools = request.get("tools", []) | ||
| return { | ||
| "system_sha256": hashlib.sha256(canonical(request.get("messages", [])[0])).hexdigest(), | ||
| "tool_names": [item.get("function", {}).get("name") for item in tools], | ||
| "tools_sha256": hashlib.sha256(canonical(tools)).hexdigest(), | ||
| "non_tools_sha256": hashlib.sha256(canonical({key: value for key, value in request.items() if key != "tools"})).hexdigest(), | ||
| } |
Comment on lines
+15
to
+23
| def credential() -> str: | ||
| value = os.environ.get("DEEPSEEK_API_KEY") | ||
| if value: | ||
| return value | ||
| path = Path(os.environ.get("DSH_HOME", Path.home() / ".dsh")) / ".credentials.yaml" | ||
| for line in path.read_text(encoding="utf-8").splitlines(): | ||
| if line.startswith("DEEPSEEK_API_KEY:"): | ||
| return line.split(":", 1)[1].strip().strip("'\"") | ||
| raise RuntimeError("未找到 DSH DeepSeek credential") |
Comment on lines
+20
to
+25
| $sourceHashes = Get-ChildItem -Recurse -File $source | ForEach-Object { | ||
| [pscustomobject]@{ Relative = [IO.Path]::GetRelativePath($source, $_.FullName); Hash = (Get-FileHash $_.FullName -Algorithm SHA256).Hash } | ||
| } | ||
| $targetHashes = Get-ChildItem -Recurse -File $target | ForEach-Object { | ||
| [pscustomobject]@{ Relative = [IO.Path]::GetRelativePath($target, $_.FullName); Hash = (Get-FileHash $_.FullName -Algorithm SHA256).Hash } | ||
| } |
Comment on lines
+24
to
+32
| def credential() -> str: | ||
| value = os.environ.get("DEEPSEEK_API_KEY") | ||
| if value: | ||
| return value | ||
| path = Path(os.environ.get("DSH_HOME", Path.home() / ".dsh")) / ".credentials.yaml" | ||
| for line in path.read_text(encoding="utf-8").splitlines(): | ||
| if line.startswith("DEEPSEEK_API_KEY:"): | ||
| return line.split(":", 1)[1].strip().strip("'\"") | ||
| raise RuntimeError("未找到 DSH DeepSeek credential") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR contributes an independent third-party replication by @NineThoughts0521 of the DeepSeek V4 Pro first-request tool-schema anchoring result on frozen Project2 V4.1b.
These runs remain separate from the maintainer's formal
n, rankings, worst scores, means, and sample index. The frozen task, verifier, evaluator, and scoring rules are unchanged.Results
All DSH runs used DeepSeek V4 Pro, reasoning
max, DSH0.1.0-rc.6, the same candidate prompt, a reset frozen workspace, and the original V4.1b evaluator.Observed descriptive contrasts:
+7+10.5+3.5Minimal-Full ablation
Minimal-Full retains the Anchored Minimal complete system condition,
complete: true,includeRuntimeContext: false, and the Standard capability roster, but removes the first-request bootstrap.The zero-cost schema gate confirms that Minimal-Full request 1 tools equal both Standard request 1 and Anchored request 2; Minimal-Full and Anchored also have equal system and request 1 non-tool hashes. Anchored request 1 exposes only
pwsh/read, then restores the full 25-tool catalog.The result supports an additional contribution from first-request tool-schema anchoring on this task and is consistent with the maintainer's original Anchored-over-Standard direction. It does not establish universal causality: each condition has one run, and the Anchored treatment also includes the catalog transition and its timing.
OpenCode disclosure
The original OpenCode run was interrupted by the outer process lifecycle after model activity had started. It is permanently retained as an unscored infrastructure failure, with its ¥0.24 cost and partial trajectory aggregate.
One explicitly approved replacement changed only the process lifecycle to a detached/background wrapper. It used OpenCode
1.18.17, direct DeepSeek,deepseek-v4-pro,--variant max, andhttps://api.deepseek.com. The replacement scored 93 and was not rerun. OpenCode remains a post-preregistered exploratory harness comparison and is excluded from the DSH mechanism ablation.Cost, evidence, and privacy
Total account cost, including the partial run, was ¥7.12. Usage recomputation using the official live CNY prices applicable to each run was ¥7.075304.
Public artifacts include scores, token/time/cost aggregates, first-request tool-catalog snapshots, catalog transitions, trajectory fingerprints, balance window hashes, infrastructure events, and an evidence manifest. Trajectory wording is treated only as a behavior fingerprint.
Raw DSH/OpenCode sessions, full reasoning/CoT, credentials, and private absolute paths remain local and Git-ignored. The public manifest records raw evidence SHA-256 values without publishing raw content.
Validation
git diff --checkpassed.The optional real ESP-IDF build was not run. DeepSWE and Terminal-Bench were not executed, so cross-task generalization remains an open question.