Summary
The #3 clean-corpus runs surfaced a real flaw in the scoring formula: the D × 3 base cost penalizes every Defensible (clean) reference, so the headline score in the Executive Dashboard becomes misleading for large reference lists. A perfectly clean 30-reference article scores only 100 − 30×3 = 10 — which reads as alarming when the article is in fact clean.
Evidence (from committed runs)
| Article |
Refs |
% Defensible |
Headline score |
| Ahmadinezhad 2024 (clean) |
30 |
83% (25/30 D) |
5 |
| Patriksson 2024 (clean) |
26 |
89% (23/26 D) |
22 |
| Madhukar 2026 (dirty, 5 refs) |
5 |
low |
54 |
A clean 30-ref article (5) scoring far lower than a dirty 5-ref article (54) is backwards as a headline integrity signal. The real calibration signal is % Defensible, which #42 adopted in the real-articles README — but the formula and the Executive Dashboard gauge still surface the misleading raw score.
The question
Should clean references incur a penalty at all? Options:
- Remove or shrink the
D × 3 term (clean refs shouldn't cost points).
- Switch the headline metric to a normalized/percentage basis (e.g., % Defensible, or score per reference) so list length doesn't dominate.
- Keep the raw score but reframe the Executive Dashboard so % Defensible is the primary number and the raw score is secondary/contextualized.
This is a prompt + report-template change, so it would ship as a vNext prompt revision (not an edit to a frozen version).
Secondary calibration note (lower priority)
In the Ahmadinezhad run, v5 gave Moderate to ref 4 (a UNICEF report — grey literature). CLAUDE.md says "do not over-flag grey literature." The Moderate was tagged "bibliographic" (couldn't fully pin the citation) rather than a fraud flag, so it's mild, but it's worth watching whether v5 is slightly conservative on confirmable grey-lit from major orgs (WHO/UNICEF/CDC). Fold into the scoring/calibration review.
Origin
Surfaced during the #3 clean-corpus addition (2026-06-19 issue-harvest sweep). References were verified real against Crossref before #42 merged; this issue tracks the scoring-methodology finding that the run exposed.
Summary
The #3 clean-corpus runs surfaced a real flaw in the scoring formula: the D × 3 base cost penalizes every Defensible (clean) reference, so the headline score in the Executive Dashboard becomes misleading for large reference lists. A perfectly clean 30-reference article scores only
100 − 30×3 = 10— which reads as alarming when the article is in fact clean.Evidence (from committed runs)
A clean 30-ref article (5) scoring far lower than a dirty 5-ref article (54) is backwards as a headline integrity signal. The real calibration signal is % Defensible, which #42 adopted in the real-articles README — but the formula and the Executive Dashboard gauge still surface the misleading raw score.
The question
Should clean references incur a penalty at all? Options:
D × 3term (clean refs shouldn't cost points).This is a prompt + report-template change, so it would ship as a vNext prompt revision (not an edit to a frozen version).
Secondary calibration note (lower priority)
In the Ahmadinezhad run, v5 gave Moderate to ref 4 (a UNICEF report — grey literature).
CLAUDE.mdsays "do not over-flag grey literature." The Moderate was tagged "bibliographic" (couldn't fully pin the citation) rather than a fraud flag, so it's mild, but it's worth watching whether v5 is slightly conservative on confirmable grey-lit from major orgs (WHO/UNICEF/CDC). Fold into the scoring/calibration review.Origin
Surfaced during the #3 clean-corpus addition (2026-06-19 issue-harvest sweep). References were verified real against Crossref before #42 merged; this issue tracks the scoring-methodology finding that the run exposed.