Skip to content

Consolidate the scoring rule into one module - #1

Merged
EvanOman merged 9 commits into
mainfrom
ddd/rule-consolidation
Aug 6, 2026
Merged

EvanOman merged 9 commits into
mainfrom
ddd/rule-consolidation

Conversation

@EvanOman

@EvanOman EvanOman commented Aug 3, 2026

Copy link
Copy Markdown
Owner

The score-to-priority/skip rule and the 60/30 value tiers were derived independently
in fourteen places. One had already diverged: tools/backfill_highlighted.py wrote
priority_score=float(result.total) with author_boost=0.0 hardcoded, so every row
it backfilled silently lost the author boost.

Rubric churn is the reason this matters — five rubric versions so far, and
scoring_strategy.py is the second-most-churned file in the repo. Each rubric change
currently requires the same edit in five to nine places, correctly.

Approach

Michael Feathers' protocol, with the gates enforced in commit order:

  1. Characterization first. The suite is committed before any production code moves,
    so the history itself proves the net predated the change. Check out 3431fad, confirm
    green; check out 56f1e96, confirm still green.
  2. The bug was pinned, not fixed. The author_boost divergence got a test asserting
    the current wrong value. It is corrected later in its own commit (97726ae) that
    deliberately updates that test, so the behavior change is visible rather than buried
    in a refactor diff.
  3. Golden master as the net. 16 articles through the full scoring pipeline, compared
    byte-for-byte. sha256:b354d0be… is unchanged at all eight commits containing it.

The net was verified, not assumed

Thresholds were mutated to confirm the tests actually catch what they claim:

Mutation Test failures
SKIP threshold 30 → 31 14
HIGH threshold 60 → 61 3
Drop author_boost from priority 25

All three caught, all reverted. A second reviewer also caught one unit section that
transcribed the rule without calling production code; it was repointed at the extracted
module, keeping the truth tables byte-identical while changing the callee.

Scope

Three more tier sites turned up after extraction (app/routers/api.py,
tools/compare_versions.py) — findable only once the rule had a name to grep against.
Fourteen sites total now route through app/domain/scoring_outcome.py.

Deliberately not done: no ORM inversion, no repository, no unit of work, no message
bus, no CQRS. The domain stays a SQLAlchemy schema.

Verification

  • just ci green: 779 passed, 30 persona deselected, 21.2s (baseline 553 tests)
  • Coverage 75%
  • Tests are hermetic — a new autouse guard fails any test that opens a socket, with its
    own test proving it blocks the LLM gateway and Readwise while still allowing subprocesses
  • The persona tier judges via a subscription-backed claude -p, so it costs no metered
    gateway budget and is excluded from just test and CI for determinism

Built in a worktree; the running reader-triage.service and its production database were
never touched.

🤖 Generated with Claude Code

Evan and others added 9 commits July 31, 2026 01:12
Adds an autouse fixture that fails any in-process network access, so the
suite can never reach Readwise or spend metered budget on the local LiteLLM
gateway. tests/test_network_guard.py proves the guard actually raises rather
than assuming it, and pins the one sanctioned exception: a subprocess, which
is how the persona tier's subscription-backed judge runs.

Adds unit/integration/golden/persona markers and per-tier just recipes. The
default run deselects persona because it depends on an external CLI being
available and is non-deterministic, not because it costs money.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Feathers-style net for the scoring-rule consolidation. Pins current behavior
at all five priority/skip write sites and the 60/30 tier sites, bugs included,
so the extraction that follows is provably behavior-preserving.

Four tiers:
  unit         boundary tables and the real cal_review tier labels
  integration  real SQLite, real write paths, all five sites
  golden       16 hand-authored articles through the real pipeline, output
               compared byte-for-byte (b354d0be)
  persona      six-article corpus judged end to end; deselected by default

The golden corpus deliberately includes the cases a naive extraction breaks:
info_score below 60 with an author boost pushing priority_score above it, and
a low scorer whose boost lifts priority while skip_recommended stays True.

test_backfill_highlighted_drops_author_boost_CHARACTERIZATION pins a real bug
on purpose: tools/backfill_highlighted.py hardcodes author_boost=0.0, so rows
it writes lose the boost every other path applies. The test asserts the wrong
value today and is updated by the later commit that fixes it.

Baseline: 553 -> 762 passing (+30 persona), coverage 75%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
app/domain/scoring_outcome.py becomes the single home for what a score means:
priority (total + author_boost), the skip decision and reason, and the 60/30
value tiers, including SQL predicates so queries filter by tier rather than
re-implementing the bands.

No call site is repointed yet — this commit only creates the module and aims
the characterization truth tables at it. The tables are byte-identical; only
the callee changed, which turns them from transcriptions of the rule into
tests of it.

Two rules are kept deliberately independent, because collapsing them is the
likeliest well-meaning mistake: priority includes the author boost, while skip
and tier read the raw total. A favoured author reorders the queue; they do not
make a contentless article valuable.

Mutation-checked against the suite: skip threshold 30->31 fails 14 tests,
tier cutoff 60->61 fails 3, dropping author_boost from priority fails 25.

This is a rules module, not a domain layer. app/models/ stays a SQLAlchemy
schema; nothing here knows about sessions or rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
app/services/scorer.py derived priority, skip_recommended and skip_reason
independently in three places. All three now come from one derive_outcome()
call, so the update branch and the insert branch cannot write different
answers for the same score.

recompute_priorities keeps its asymmetry on purpose: it calls derive_priority
alone and still does not touch the stored skip flags. Re-deriving skip there
would silently rewrite decisions the scoring run already made, so the
narrower call is the correct one and is now explicit rather than incidental.

Behavior unchanged: 779 passing, golden master b354d0be unmoved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
backfill_v5 re-derived priority, skip and the High-tier comparison inline;
it now shares the one implementation, so a reweighting run and a scoring run
agree by construction rather than by coincidence.

backfill_highlighted is repointed WITHOUT fixing its divergence. It still
feeds the rule a zero author boost, so rows it writes still lose the boost
every other path applies — but the zero is now an explicit argument with a
comment naming it, instead of a hardcoded field that read like a decision.
test_backfill_highlighted_drops_author_boost_CHARACTERIZATION still passes
against the wrong values, which is the proof this commit changed nothing.
The fix is the next commit.

Behavior unchanged: 779 passing, golden master b354d0be unmoved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two podcast routers each re-implemented the 60/30 bands twice — once as
an if/elif filter and once as three count queries — for four copies between
them. All four now call tier_criteria(), which returns SQL predicates so the
filtering still happens in the database.

An unrecognised tier string yields no criteria, which leaves the query
unfiltered exactly as the old if/elif fall-through did; verified that
where() with zero criteria compiles to the unfiltered statement.

Podcast totals stay on their own scale (four 0-25 dimensions summed) while
sharing the article cutoffs. Only the cutoffs were duplicated, so only the
cutoffs are shared.

Behavior unchanged: 779 passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cal_review's two tier-label helpers each carried their own copy of the
60/30 bands; they now ask value_tier() and keep only the presentation — which
word, which color. digest stops defining HIGH_VALUE_THRESHOLD a second time
and re-exports the shared one, so app.services.digest.HIGH_VALUE_THRESHOLD
still imports at the same name with the same float value.

Both rough edges in cal_review are preserved on purpose, because this commit
is a refactor and not a fix: int(score) truncation stays, so a NaN score
still raises ValueError and an infinite one still raises OverflowError rather
than degrading to "N/A". Characterization tests pin those exact exception
types.

Behavior unchanged: 779 passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This is the behavior change the preceding refactor was built to make safe.

tools/backfill_highlighted.py hardcoded author_boost=0.0, so an article it
backfilled ranked below an identical article by the same author that happened
to go through the scorer. It now looks the boost up through the same lookup
every other write path uses, so which code path wrote a row no longer changes
where the article ranks.

test_backfill_highlighted_drops_author_boost_CHARACTERIZATION pinned the wrong
values on purpose and is inverted here, in the same commit, and renamed to
test_backfill_highlighted_applies_author_boost. It now asserts that the
backfill and the scorer produce the same priority for the same article, which
is the property actually worth protecting. Verified it fails when the fix is
reverted.

Not backfilled: rows already written with a zero boost keep it until
ArticleScorer.recompute_priorities runs, which is the existing mechanism for
exactly this and needs no new migration.

779 passing; golden master b354d0be unmoved, as expected — it covers the
scorer pipeline, which this commit does not touch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The original survey named six tier blocks. Grepping the finished refactor
found three more that implement the same 60/30 bands: the article tier filter
and the three article tier counts in app/routers/api.py, and the tier
histogram in tools/compare_versions.py. The article filter is the exact
counterpart of the podcast one already repointed, so leaving it would have
meant the rule still had two homes while the grep suggested one.

tier_criteria now takes SQLColumnExpression rather than ColumnElement, which
is what lets a mapped column (ArticleScore.info_score, an
InstrumentedAttribute) and a computed one (score_total_expr()) both pass a
strict type check without a cast or a suppression.

Not repointed: tools/cal_report.py:317 compares an engagement rate against
60. Same number, unrelated quantity — sharing the constant would couple two
things that only look alike.

779 passing, ty clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@EvanOman
EvanOman merged commit 4169fa8 into main Aug 6, 2026
3 checks passed
EvanOman added a commit that referenced this pull request Nov 3, 2026
Preserving the commit history deliberately: the ordering is the evidence that the characterization suite predated the refactor it protects.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant