thresher

A generic self-learning classifier + decision nudger. Separates grain from chaff — e.g. hard from trivial tasks, important from ignorable — using a weighted lexicon that learns from feedback. No model, no training data beyond the decisions you already make; just counting and trust thresholds.

Generalized from a subagent-routing classifier, this is the reusable core: you provide the class names, the signal words, and the meaning of over/under, and thresher handles the learning and the nudge.

How it works

Classification — a weighted-count lexicon:

• STRONG signals (weight 2) + WEAK signals (weight 1). An item is classified into the positive class when its summed score >= threshold.
• Tokens are counted, not matched by a model — so a single generic word never forces the positive class; you need two weak words or one strong one.

Confidence + ambiguity — every classification returns a Prediction with label, score, confidence, and ambiguous. Items too close to the decision boundary are flagged unknown (needs human review) rather than forced into a class. Use predict() for the full result; classify() stays backward-compatible.

Self-learning lexicon — grows from stated decisions:

• Every time a decision carries a declared class (e.g. "this was a hard task"), the tokens in that task's text increment evidence for that class.
• A token is promoted into the effective signal set once it robustly co-occurs with a class (count >= min_count AND ratio >= min_ratio).
• The classifier improves automatically as you keep making decisions — the signal vocabulary calibrates to your vocabulary.
• Optional decay (cfg.decay > 0): old evidence fades each update, so stale signals stop firing when your domain shifts.

Evaluation — measure whether it actually learns:

• evaluate() computes accuracy / precision / recall / F1 against stated labels (train/test split supported).
• learning_curve() shows how metrics improve as the lexicon trains on more evidence — proof the self-learning is working.
• adaptive_threshold() sweeps candidate thresholds and returns the one that maximizes F1 on your data.

The nudge — a feedback pass:

• Detects over-escalation (stated light/negative but a heavy class chosen) and under-escalation (stated positive but the default chosen).
• Renders a short, actionable lessons note (e.g. routing-lessons.md) that a downstream agent reads at the start of its next session. With graded=True it appends a severity grade (🟢 low / 🟡 moderate / 🔴 high) based on the mismatch rate.

Install

pip install -e .
# or: uv sync

Usage

from thresher import ClassConfig, FeedbackConfig, Lexicon, classify

cfg = ClassConfig(name="My problem", pos_class="hard", neg_class="light",
                  strong_words=["architectural"], weak_words=["parsing"],
                  neg_words=["trivial"])
lex = Lexicon(cfg, "~/lexicon.json")
lex.update_from_records(records)   # self-learn from stated decisions
print(classify("refactor the parser", cfg, lex))

Or via CLI:

# classify one item
thresher classify --config examples/routing_cfg.py --text "refactor the parser"

# learn from a JSONL of stated decisions
thresher learn --config examples/routing_cfg.py \
               --records examples/records.jsonl --lexicon /tmp/lex.json

# analyze decisions and write a lessons note (the nudge)
thresher analyze --config examples/routing_cfg.py \
                 --records examples/records.jsonl \
                 --lexicon /tmp/lex.json --output /tmp/lessons.md

Records format (JSONL)

Each line:

{"text": "the item to classify",
 "stated_class": "hard",
 "chosen_class": "advisor",
 "timestamp": "2026-08-07T09:49:00Z",
 "source": ""}          // "backfill" = machine-inferred; not used for learning

The example

examples/routing_cfg.py is the original use case (subagent routing): decide if a planning task is hard (use the heavy advisor) or light (use the default consultant). It comes with examples/records.jsonl showing over/under-escalation that the analyzer detects.

License

MIT