Repository navigation
Consolidate the scoring rule into one module - #1
Merged
Merged
Conversation
Adds an autouse fixture that fails any in-process network access, so the suite can never reach Readwise or spend metered budget on the local LiteLLM gateway. tests/test_network_guard.py proves the guard actually raises rather than assuming it, and pins the one sanctioned exception: a subprocess, which is how the persona tier's subscription-backed judge runs. Adds unit/integration/golden/persona markers and per-tier just recipes. The default run deselects persona because it depends on an external CLI being available and is non-deterministic, not because it costs money. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Feathers-style net for the scoring-rule consolidation. Pins current behavior
at all five priority/skip write sites and the 60/30 tier sites, bugs included,
so the extraction that follows is provably behavior-preserving.
Four tiers:
unit boundary tables and the real cal_review tier labels
integration real SQLite, real write paths, all five sites
golden 16 hand-authored articles through the real pipeline, output
compared byte-for-byte (b354d0be)
persona six-article corpus judged end to end; deselected by default
The golden corpus deliberately includes the cases a naive extraction breaks:
info_score below 60 with an author boost pushing priority_score above it, and
a low scorer whose boost lifts priority while skip_recommended stays True.
test_backfill_highlighted_drops_author_boost_CHARACTERIZATION pins a real bug
on purpose: tools/backfill_highlighted.py hardcodes author_boost=0.0, so rows
it writes lose the boost every other path applies. The test asserts the wrong
value today and is updated by the later commit that fixes it.
Baseline: 553 -> 762 passing (+30 persona), coverage 75%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
app/domain/scoring_outcome.py becomes the single home for what a score means: priority (total + author_boost), the skip decision and reason, and the 60/30 value tiers, including SQL predicates so queries filter by tier rather than re-implementing the bands. No call site is repointed yet — this commit only creates the module and aims the characterization truth tables at it. The tables are byte-identical; only the callee changed, which turns them from transcriptions of the rule into tests of it. Two rules are kept deliberately independent, because collapsing them is the likeliest well-meaning mistake: priority includes the author boost, while skip and tier read the raw total. A favoured author reorders the queue; they do not make a contentless article valuable. Mutation-checked against the suite: skip threshold 30->31 fails 14 tests, tier cutoff 60->61 fails 3, dropping author_boost from priority fails 25. This is a rules module, not a domain layer. app/models/ stays a SQLAlchemy schema; nothing here knows about sessions or rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
app/services/scorer.py derived priority, skip_recommended and skip_reason independently in three places. All three now come from one derive_outcome() call, so the update branch and the insert branch cannot write different answers for the same score. recompute_priorities keeps its asymmetry on purpose: it calls derive_priority alone and still does not touch the stored skip flags. Re-deriving skip there would silently rewrite decisions the scoring run already made, so the narrower call is the correct one and is now explicit rather than incidental. Behavior unchanged: 779 passing, golden master b354d0be unmoved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
backfill_v5 re-derived priority, skip and the High-tier comparison inline; it now shares the one implementation, so a reweighting run and a scoring run agree by construction rather than by coincidence. backfill_highlighted is repointed WITHOUT fixing its divergence. It still feeds the rule a zero author boost, so rows it writes still lose the boost every other path applies — but the zero is now an explicit argument with a comment naming it, instead of a hardcoded field that read like a decision. test_backfill_highlighted_drops_author_boost_CHARACTERIZATION still passes against the wrong values, which is the proof this commit changed nothing. The fix is the next commit. Behavior unchanged: 779 passing, golden master b354d0be unmoved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two podcast routers each re-implemented the 60/30 bands twice — once as an if/elif filter and once as three count queries — for four copies between them. All four now call tier_criteria(), which returns SQL predicates so the filtering still happens in the database. An unrecognised tier string yields no criteria, which leaves the query unfiltered exactly as the old if/elif fall-through did; verified that where() with zero criteria compiles to the unfiltered statement. Podcast totals stay on their own scale (four 0-25 dimensions summed) while sharing the article cutoffs. Only the cutoffs were duplicated, so only the cutoffs are shared. Behavior unchanged: 779 passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cal_review's two tier-label helpers each carried their own copy of the 60/30 bands; they now ask value_tier() and keep only the presentation — which word, which color. digest stops defining HIGH_VALUE_THRESHOLD a second time and re-exports the shared one, so app.services.digest.HIGH_VALUE_THRESHOLD still imports at the same name with the same float value. Both rough edges in cal_review are preserved on purpose, because this commit is a refactor and not a fix: int(score) truncation stays, so a NaN score still raises ValueError and an infinite one still raises OverflowError rather than degrading to "N/A". Characterization tests pin those exact exception types. Behavior unchanged: 779 passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This is the behavior change the preceding refactor was built to make safe. tools/backfill_highlighted.py hardcoded author_boost=0.0, so an article it backfilled ranked below an identical article by the same author that happened to go through the scorer. It now looks the boost up through the same lookup every other write path uses, so which code path wrote a row no longer changes where the article ranks. test_backfill_highlighted_drops_author_boost_CHARACTERIZATION pinned the wrong values on purpose and is inverted here, in the same commit, and renamed to test_backfill_highlighted_applies_author_boost. It now asserts that the backfill and the scorer produce the same priority for the same article, which is the property actually worth protecting. Verified it fails when the fix is reverted. Not backfilled: rows already written with a zero boost keep it until ArticleScorer.recompute_priorities runs, which is the existing mechanism for exactly this and needs no new migration. 779 passing; golden master b354d0be unmoved, as expected — it covers the scorer pipeline, which this commit does not touch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The original survey named six tier blocks. Grepping the finished refactor found three more that implement the same 60/30 bands: the article tier filter and the three article tier counts in app/routers/api.py, and the tier histogram in tools/compare_versions.py. The article filter is the exact counterpart of the podcast one already repointed, so leaving it would have meant the rule still had two homes while the grep suggested one. tier_criteria now takes SQLColumnExpression rather than ColumnElement, which is what lets a mapped column (ArticleScore.info_score, an InstrumentedAttribute) and a computed one (score_total_expr()) both pass a strict type check without a cast or a suppression. Not repointed: tools/cal_report.py:317 compares an engagement rate against 60. Same number, unrelated quantity — sharing the constant would couple two things that only look alike. 779 passing, ty clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
EvanOman
added a commit
that referenced
this pull request
Nov 3, 2026
Preserving the commit history deliberately: the ordering is the evidence that the characterization suite predated the refactor it protects.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The score-to-priority/skip rule and the 60/30 value tiers were derived independently
in fourteen places. One had already diverged:
tools/backfill_highlighted.pywrotepriority_score=float(result.total)withauthor_boost=0.0hardcoded, so every rowit backfilled silently lost the author boost.
Rubric churn is the reason this matters — five rubric versions so far, and
scoring_strategy.pyis the second-most-churned file in the repo. Each rubric changecurrently requires the same edit in five to nine places, correctly.
Approach
Michael Feathers' protocol, with the gates enforced in commit order:
so the history itself proves the net predated the change. Check out
3431fad, confirmgreen; check out
56f1e96, confirm still green.author_boostdivergence got a test assertingthe current wrong value. It is corrected later in its own commit (
97726ae) thatdeliberately updates that test, so the behavior change is visible rather than buried
in a refactor diff.
byte-for-byte.
sha256:b354d0be…is unchanged at all eight commits containing it.The net was verified, not assumed
Thresholds were mutated to confirm the tests actually catch what they claim:
author_boostfrom priorityAll three caught, all reverted. A second reviewer also caught one unit section that
transcribed the rule without calling production code; it was repointed at the extracted
module, keeping the truth tables byte-identical while changing the callee.
Scope
Three more tier sites turned up after extraction (
app/routers/api.py,tools/compare_versions.py) — findable only once the rule had a name to grep against.Fourteen sites total now route through
app/domain/scoring_outcome.py.Deliberately not done: no ORM inversion, no repository, no unit of work, no message
bus, no CQRS. The domain stays a SQLAlchemy schema.
Verification
just cigreen: 779 passed, 30 persona deselected, 21.2s (baseline 553 tests)own test proving it blocks the LLM gateway and Readwise while still allowing subprocesses
claude -p, so it costs no meteredgateway budget and is excluded from
just testand CI for determinismBuilt in a worktree; the running
reader-triage.serviceand its production database werenever touched.
🤖 Generated with Claude Code