chore(graph): activate 51 evidence packs that were only ever waiting on the scorer - #489
Merged
Merged
Conversation
…on the scorer These packs have been sitting in `draft` for weeks — the oldest for 22 days — not because they were incomplete, but because `forgeplan score` reported them as worth 0.0. That was the leaf-EVID defect (#325): the scorer asked a pack for *its* evidence, found none (a pack has no packs), and returned zero. Activating them under the old binary would have stamped that zero into the graph as if it were a measurement. v0.37.0 fixes the scorer, so this is the first moment the activation says something true. Verified with the published 0.37.0 binary before touching anything: all 51 validate PASS, 40 score >= 0.7, and the three that score low do so correctly — EVID-139 Weakens CL3 = 0.30 EVID-141 Weakens CL3 = 0.40 EVID-146 Refutes CL3 = 0.00 — honest weakening and refuting evidence, which is exactly what those verdicts are supposed to produce. Not rubber-stamped: each was read before being activated. Graph after: draft 152 -> 101, active 212 -> 263, health verdict "unhealthy — multiple critical signals" -> "needs attention". The two remaining at-risk artifacts are real debt, not bookkeeping, and are deliberately left alone here: PRD-005 R_eff 0.00 — its only pack EVID-010 is deprecated; it needs a new measurement, not a scoring change RFC-004 R_eff 0.10 — EVID-035 declares no structured fields, so it scores CL0. This is the debt the v0.37.0 changelog predicted would surface after re-scoring. Refs: #325, PRD-086, v0.37.0
Both files had drifted into confidently stating wrong facts, which is worse than saying nothing. TODO.md announced `Current: v0.34.0` while three releases had shipped past it (v0.35/36/37), and still carried an "in-flight to v0.31.0, PR #277 open, awaiting CI" section — v0.31.0 landed four months ago. It was structured as a reverse-chronological release log that duplicated CHANGELOG.md and was only ever touched at release time, so it fell behind the moment two releases went out without updating it. docs/ROADMAP.md was generated 2026-04-11 after v0.18.0 — five months and ~19 minor versions stale. Its baselines were off by roughly 2x: 193 artifacts (actual 437), 76 CLI commands (actual 82), 1940 tests (actual 3331). MCP tool count 73 was the single number that still held. Every figure in both files now carries its measurement source rather than being copied forward. Gaps were re-checked against the code, not restated: verified DONE website (385 markdown files under website/src/content), CI gates (validate --ci, health --ci --fail-on --strict — shipped April, listed as an open HIGH-impact gap since), semantic search in all 5 release binaries since v0.35.0, incremental embed (v0.36.0) verified OPEN forgeplan doctor/links/diff (absent from the Commands enum; `plugins doctor` exists and is a narrower thing), crates.io (GET /api/v1/crates/forgeplan -> 404), Docker (no Dockerfile), Tauri desktop, VS Code ext, Linear/Jira, pluggable storage drivers (RFC-003) unverified left labelled as such rather than guessed TODO.md is now a priority list rather than a release log: health signals first (the cheapest next steps, since the tool already named the problem and the command), then a tiered backlog, then a pointer at the live GitHub issue list instead of a copy that would go stale immediately. It ends with the recipe for re-verifying itself, so the next person does not repeat this archaeology. Worth flagging beyond the file edits: EVID-033/034/035 score CL0 for missing structured fields — the same defect class fixed per-instance in April on EVID-015/025/026/027. Recurring on fresh IDs five months later says the fix was per-instance and the class is still open; a validate-time gate would close it properly. Refs: v0.37.0 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine PRDs described work that shipped releases ago and were never
activated. This is bookkeeping debt, not backlog: leaving them in `draft`
made "101 drafts" overstate the real backlog by roughly a third.
Each was confirmed against something concrete before activating -- a live
MCP tool matching the PRD's own name, or a direct code/website grep hit --
never from the title sounding familiar:
PRD-054 activity log forgeplan_activity, _activity_stats
PRD-056 phase machine forgeplan_phase, _phase_advance
PRD-065 playbook runtime forgeplan_playbook_{list,run,show,validate}
PRD-066 ingest engine forgeplan_ingest
PRD-067 plugin detection forgeplan_plugins_{doctor,info,list}
PRD-070 CLI parity all named commands present
PRD-072 subprocess dispatch playbook/dispatch/*.rs cites ADR-010 by name
PRD-073 ADR-003 file-first PR #230, 3 evidence packs
PRD-080 blog foundation website/src/components/blog/* exist
R_eff for all nine is above zero, so RED LINE #7 holds. PRD-080 sits at
0.20 because of a CL penalty inherited from PRD-079 through a `based_on`
edge -- its own evidence is CL3 at 1.0. The score is reporting a weak link
upstream, not missing work, which is the cascade doing its job.
Two candidates were held back rather than activated:
PRD-055 R_eff 0.0, no evidence at all. Activating it would violate
RED LINE #7 -- the work shipped (forgeplan_restore and
forgeplan_undo_last both exist), but the graph has nothing
backing that claim.
PRD-071 same. The triage report said EVID-086 existed but was never
linked; checked directly -- `forgeplan get EVID-086` returns
"not found". The pack does not exist, so there is nothing to
link. That correction matters: the fix here is writing
evidence, not repairing an edge.
PRD-078 was flagged and deliberately left alone: its code is on `dev`, but
EVID-139 (a 2-agent adversarial audit) records an unresolved HIGH
concurrency finding and a HIGH scope gap, and EVID-141 records the latency
NFR being missed. R_eff 0.30 is that contest showing up honestly. Activating
it would assert something the evidence disputes.
PROB-090 -- the graph's only orphan -- is now linked. Its own body already
named PRD-083 as where it was found; that was prose, never an edge. Also
linked to PROB-085, which is the same defect class (unsynchronised mutation
of process-global state racing under the parallel test runner: cwd there,
PATH here), so whoever fixes one finds the other.
draft 101 -> 92, active 263 -> 272, orphans 1 -> 0
Refs: #454, PROB-090, v0.37.0
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Artifact-only. 51 files, each a single
status: draft → activeline.Why now and not earlier
These packs sat in
draftfor weeks — the oldest 22 days — becauseforgeplan scorereported them as worth 0.0. That was the leaf-EVID defect (#325): the scorer asked a pack for its own evidence, found none (a pack has no packs), and returned zero. Activating them under the old binary would have written that zero into the graph as though it were a measurement.v0.37.0 fixes the scorer. Same artifact, both binaries:
This is the first moment the activation says something true.
Not rubber-stamped
Checked all 51 with the published 0.37.0 binary before touching anything — every one validates PASS. Score distribution: 40 at ≥0.7 (30 of them exactly 1.00), and the three low ones are low correctly:
WeakensCL3 — an honest adversarial-audit findingWeakensCL3 — sameRefutesCL3 — a refutation scoring zero is the intended readingEight more initially read as "no score" in my sweep; that was my grep, not the tool — they carry child evidence and take the normal weakest-link path, all at 1.00.
Effect
Deliberately not touched
Two at-risk artifacts remain, and both are real debt rather than bookkeeping:
R_eff 0.00. Its only evidence pack EVID-010 is deprecated, so there is genuinely nothing current supporting it. Needs a new measurement, not a scoring change.R_eff 0.10. EVID-035 declares no structured fields and therefore scores CL0. This is precisely the debt the v0.37.0 changelog said would surface once packs were re-scored honestly.Also filed while doing this: #488 —
forgeplan scoretells a refuting leaf pack to add structured fields it already has. The scoring is right; only the hint is wrong. Found on EVID-146 above.