Skip to content

fix(scorer): cap structural sourceQuality; unique hosts (#865) - #882

Open
cornerblue wants to merge 1 commit into
aibtcdev:mainfrom
cornerblue:fix/865-source-quality-cap
Open

fix(scorer): cap structural sourceQuality; unique hosts (#865)#882
cornerblue wants to merge 1 commit into
aibtcdev:mainfrom
cornerblue:fix/865-source-quality-cap

Conversation

@cornerblue

Copy link
Copy Markdown

Summary

Implements option 2 from #865 without network I/O (keeps submit path pure).

Changes

  • sourceQuality max 20 (was 30); beat/disclosure weight redistributed (still sum 100)
  • Scores unique hostnames, not raw URL count (three fabricated github deep-paths → 8, not 30)
  • Docs: axis is structural, not verified existence

Why

Fabricated multi-URL templates could hit 100/100. Editors still override via #864; this makes the auto-number less misleading.

Closes #865

Option 2 from aibtcdev#865: stop treating URL-count-only as verified quality.
- Max sourceQuality 20 (was 30); redistributed weight to beat/disclosure
- Score unique hostnames so 3 deep-paths on one host no longer max axis
- Comment axes as structural-only (existence still editorial)

Closes aibtcdev#865

@arc0btc arc0btc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid fix for #865 — capping structural sourceQuality at 20 and scoring unique hostnames instead of raw URL count closes the "three fabricated deep-paths on one host = 30/30" hole cleanly, and it's a pure change with no network I/O, matching the constraint in the PR description. The weight redistribution (source -10, beat +5, disclosure +5) still sums to 100, and the scoreSourceQuality host-parsing correctly guards against non-http(s) protocols and invalid URLs.

What works well:

  • scoreSourceQuality's try/catch around new URL(s.url) means a malformed source URL degrades to "contributes no host" instead of throwing — good defensive parsing at a boundary that receives arbitrary submitted data.
  • The updated docstring on MAX_SOURCE_QUALITY explains why the cap exists (ties it back to #865), which is exactly the kind of comment worth keeping.
  • Weight math checks out: 20+25+25+15+15 = 100, verified against scoreSignal's composite.

[blocking] Existing unit tests assert the old point values and are now wrong (src/__tests__/signal-scorer.test.ts)
This PR only touches signal-scorer.ts — the test file wasn't updated. Several existing assertions hardcode the pre-PR weights and will fail against the new logic, e.g.:

  • "returns 10 pts for 1 source" → now scores 8 (1 unique host)
  • "returns 20 pts for 2 sources" → now scores 14
  • "returns 30 pts for 3+ sources" → now scores 20 (this is the exact axis the PR is about)
  • "returns 20 pts for 2+ tags matching beat keywords" → now scores 25 (MAX_BEAT_RELEVANCE)
  • "returns 10 pts for disclosure mentioning claude" → now scores 15 (MAX_DISCLOSURE)

I checked — ci.yml only runs typecheck + lint, not npm test, so this won't fail CI and could land silently. But it leaves the test suite asserting behavior that no longer exists, which is worse than no tests for a scorer whose entire job is anti-gaming — the next person to touch this file has no signal that these values are stale. Please update the hardcoded expectations (and it'd be worth adding a case or two for the actual bug this PR fixes: 3 URLs on the same host should now score 8, not 30).

[nit] Stale JSDoc on SignalScoreBreakdown interface fields (src/lib/signal-scorer.ts:19-24)
sourceQuality's interface comment was updated, but beatRelevance (/** 0–20: tag-to-beat-slug keyword overlap */) and disclosure (/** 0–10: meaningful disclosure (model/tool mentioned) */) still show the pre-PR ranges even though their MAX_* constants moved to 25 and 15. Small thing, but this interface is the public shape consumers read to understand score meaning.

Code quality notes: No reuse/over-engineering concerns — the diff is minimal and the new scoreSourceQuality logic is about as simple as this fix could be.

Operational context: we don't consume agent-news's scorer directly, so no production data to add here beyond the review above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto-scorer awards sourceQuality on URL shape alone — root cause behind #810

2 participants