fix(scorer): cap structural sourceQuality; unique hosts (#865) - #882
fix(scorer): cap structural sourceQuality; unique hosts (#865)#882cornerblue wants to merge 1 commit into
Conversation
Option 2 from aibtcdev#865: stop treating URL-count-only as verified quality. - Max sourceQuality 20 (was 30); redistributed weight to beat/disclosure - Score unique hostnames so 3 deep-paths on one host no longer max axis - Comment axes as structural-only (existence still editorial) Closes aibtcdev#865
arc0btc
left a comment
There was a problem hiding this comment.
Solid fix for #865 — capping structural sourceQuality at 20 and scoring unique hostnames instead of raw URL count closes the "three fabricated deep-paths on one host = 30/30" hole cleanly, and it's a pure change with no network I/O, matching the constraint in the PR description. The weight redistribution (source -10, beat +5, disclosure +5) still sums to 100, and the scoreSourceQuality host-parsing correctly guards against non-http(s) protocols and invalid URLs.
What works well:
scoreSourceQuality'stry/catcharoundnew URL(s.url)means a malformed source URL degrades to "contributes no host" instead of throwing — good defensive parsing at a boundary that receives arbitrary submitted data.- The updated docstring on
MAX_SOURCE_QUALITYexplains why the cap exists (ties it back to #865), which is exactly the kind of comment worth keeping. - Weight math checks out: 20+25+25+15+15 = 100, verified against
scoreSignal's composite.
[blocking] Existing unit tests assert the old point values and are now wrong (src/__tests__/signal-scorer.test.ts)
This PR only touches signal-scorer.ts — the test file wasn't updated. Several existing assertions hardcode the pre-PR weights and will fail against the new logic, e.g.:
"returns 10 pts for 1 source"→ now scores 8 (1 unique host)"returns 20 pts for 2 sources"→ now scores 14"returns 30 pts for 3+ sources"→ now scores 20 (this is the exact axis the PR is about)"returns 20 pts for 2+ tags matching beat keywords"→ now scores 25 (MAX_BEAT_RELEVANCE)"returns 10 pts for disclosure mentioning claude"→ now scores 15 (MAX_DISCLOSURE)
I checked — ci.yml only runs typecheck + lint, not npm test, so this won't fail CI and could land silently. But it leaves the test suite asserting behavior that no longer exists, which is worse than no tests for a scorer whose entire job is anti-gaming — the next person to touch this file has no signal that these values are stale. Please update the hardcoded expectations (and it'd be worth adding a case or two for the actual bug this PR fixes: 3 URLs on the same host should now score 8, not 30).
[nit] Stale JSDoc on SignalScoreBreakdown interface fields (src/lib/signal-scorer.ts:19-24)
sourceQuality's interface comment was updated, but beatRelevance (/** 0–20: tag-to-beat-slug keyword overlap */) and disclosure (/** 0–10: meaningful disclosure (model/tool mentioned) */) still show the pre-PR ranges even though their MAX_* constants moved to 25 and 15. Small thing, but this interface is the public shape consumers read to understand score meaning.
Code quality notes: No reuse/over-engineering concerns — the diff is minimal and the new scoreSourceQuality logic is about as simple as this fix could be.
Operational context: we don't consume agent-news's scorer directly, so no production data to add here beyond the review above.
Summary
Implements option 2 from #865 without network I/O (keeps submit path pure).
Changes
sourceQualitymax 20 (was 30); beat/disclosure weight redistributed (still sum 100)Why
Fabricated multi-URL templates could hit 100/100. Editors still override via #864; this makes the auto-number less misleading.
Closes #865