fix(novelty): routing estimate and permanent-error stop

dokimi/changelog.d/20261113_150000_routing-estimate-permanent-stop.md

@@ -0,0 +1,4 @@
+### Fixed
+
+- `skill routing --estimate-only` and `catalog routing --estimate-only` size the selection prompt at 2.6 characters per token and a reply at 150 tokens (it was chars / 4 and 65, about 1.55x below measured bills), and print both allowances. (#676)
+- `skill routing` and `catalog routing` stop at the first authentication, permission, unknown-model, unsupported-parameter or no-credentials error (with `claude-cli`, also not logged in, a usage limit or no `claude` binary) instead of failing every remaining call; the result is partial, with `reason` `permanent_error` and the error, and the command exits 1 (a `skill routing` run whose every call failed exited 0 with status `ok`). Any other failure, such as a request too long for the model, is still one failed row. (#677)

dokimi/docs/skill-novelty.md

@@ -729,11 +729,14 @@ others are named on stderr.
 
 `--estimate-only` prints, as JSON, the calls, input and output tokens and the list-price `expected_usd` the
 `--model` arm would make, and makes no model call. Without `--model` the run is lexical and the bill
-is zero. Input is the selection prompt sized at chars / 4, so it is a projection, not a count.
-Output is `output_tokens_per_call` (65) times the calls: the average a measured routing run billed per
-selection call (360 calls drew 23.3k output tokens). The prompt asks for one `{"pick": N}` object, about
-10 tokens, which underestimated that run 6.5 times, so the estimate does not use it. Another model, or a
-reasoning model, can bill more or less.
+is zero. Input is the selection prompt sized at `chars_per_input_token` (2.6) characters per token, counted
+locally with no tokenizer, so it is a projection, not a count. Output is `output_tokens_per_call` (150) times
+the calls: one flat allowance, inside the 114 to 193 tokens a call that Sonnet 5.5 and Haiku 5.5 billed.
+The prompt asks for one `{"pick": N}` object, about 10 tokens, which underestimated a measured run 6.5
+times, so the estimate does not use it. Both allowances come from one measured catalog (35 skills, 90
+queries, Bedrock): with chars / 4 and 65 tokens the estimate was about 1.55x below the bills there. Another
+model, a catalog of other text, or a reasoning model can bill more or less, so read the figure as an
+estimate, not a bound. The command prints both allowances.
 
 Two numbers, reported side by side and never merged:
 
@@ -824,11 +827,13 @@ so count the queries first, or pass `--max-usd` to stop the run at a spend limit
 A stored response in `DOKIMI_RESPONSE_CACHE` is replayed without a call, so a rerun can make
 fewer calls than the printed count.
 
+**A permanent provider error that every call would meet stops a routing run the way the cap does.** An authentication or permission error, an unknown model ID (`not found`), a parameter the provider does not take (every call sends the same ones) or missing credentials (with `claude-cli`: not logged in, a usage limit, no `claude` binary) would fail every later call alike, so the run stops at that call, keeps the rows made before it and writes the partial result below, with `"reason": "permanent_error"` and the error text as the message (`--max-usd` is not involved; the exit line gives `calls_done` of `calls_planned`). Any other failure is one failed row and the run goes on: a refusal, a request one query makes too long, an unparseable reply, an error the client does not recognise, a transient error that ran out of retries.
+
 **A routing run stopped by `--max-usd` keeps what it paid for.** `skill routing --model`, single
 artifact or `catalog routing`, writes the result of the calls made before the cap (or a call with no
 known price) stopped it, prints the summary and `cost` lines, and still exits non-zero with the
 cap message and the path of the file. The file carries
-`"partial": {"reason": "spend_cap", "calls_done": N, "calls_planned": M, "message": ...}` at the
+`"partial": {"reason": "spend_cap", "calls_done": N, "calls_planned": M, "message": ...}` (`"permanent_error"` for the stop above) at the
 top level (and under the model arm), and no `partial` key when the run finished. The record's
 `status` is then `"partial"` instead of `"ok"` (a single run; an all-vs-all file carries `status`
 only when partial), so a script that checks `status == "ok"` does not take a prefix of the run for

dokimi/dokimi/commands/novelty_routing_report.py

@@ -62,7 +62,7 @@ def estimate_routing_usd(model: str, provider: str, prompts: list[int],
     priced on their own, then summed and multiplied by the repeats. ``None`` when a call has no list price.
     """
     from dokimi.llm_pricing import list_cost_usd
-    from dokimi.novelty.routing import SELECTION_REPLY_TOKENS
+    from dokimi.novelty.routing_estimate import SELECTION_REPLY_TOKENS
 
     total = 0.0
     for prompt in prompts:
@@ -315,7 +315,11 @@ def drop_queries_left_out(queries_by_skill: dict[str, list[str]], field: list[Ca
 def _print_estimate(args: argparse.Namespace, candidates: list, sent: list[str],
                     estimate: dict) -> None:
     """The estimate as JSON, with the list-price USD of the calls ``sent`` makes; nothing is sent."""
-    from dokimi.novelty.routing import SELECTION_REPLY_TOKENS, selection_prompt_tokens
+    from dokimi.novelty.routing_estimate import (
+        SELECTION_CHARS_PER_TOKEN,
+        SELECTION_REPLY_TOKENS,
+        selection_prompt_tokens,
+    )
 
     usd: float | None = 0.0
     if args.model and estimate["n_calls"]:
@@ -325,9 +329,11 @@ def _print_estimate(args: argparse.Namespace, candidates: list, sent: list[str],
                       "provider": args.provider if args.model else None,
                       "n_candidates": len(candidates), **estimate,
                       "expected_usd": usd}, indent=2))
-    print("Estimate only, nothing was measured. Input is the selection prompt at chars / 4 per "
-          f"call; output is {SELECTION_REPLY_TOKENS} tokens a call, the measured average of a "
-          "routing run (output_tokens_per_call). Another model, or a reasoning model, can differ.",
+    print("Estimate only, nothing was measured. Input is the selection prompt at "
+          f"{SELECTION_CHARS_PER_TOKEN} characters per token (chars_per_input_token); output is "
+          f"{SELECTION_REPLY_TOKENS} tokens a call (output_tokens_per_call), inside the 114 to 193 "
+          "that two models billed. With chars / 4 and 65 the estimate was about 1.55x below the "
+          "bills. Another model, a tokenizer or a reasoning model can differ.",
           file=sys.stderr)
     if args.model and estimate["n_calls"] and usd is None:
         print(f"expected_usd is null: {args.model} has no list price in the litellm price map",
@@ -339,7 +345,7 @@ def print_routing_estimate(args: argparse.Namespace, candidates: list, queries:
 
     No client is built. Without ``--model`` the run is lexical only, so the bill is zero.
     """
-    from dokimi.novelty.routing import selection_estimate
+    from dokimi.novelty.routing_estimate import selection_estimate
 
     # Without --model there is no selector, so no call is made, whatever --reps says.
     estimate = selection_estimate(candidates, queries,
@@ -354,7 +360,8 @@ def print_all_vs_all_estimate(args: argparse.Namespace, field: list[Candidate],
     The model arm sends each query with a content word, every skill's queries with the whole field
     on each call, so those are what is counted. ``n_queries`` is every query the run would route.
     """
-    from dokimi.novelty.routing import selection_estimate, tokenize
+    from dokimi.novelty.routing import tokenize
+    from dokimi.novelty.routing_estimate import selection_estimate
 
     sent = [query for queries in queries_by_skill.values() for query in queries if tokenize(query)]
     estimate = selection_estimate(field, sent, repeats=args.repeats if args.model else 0)

dokimi/dokimi/llm_cli.py

@@ -33,7 +33,7 @@
   subscription is charged; it is kept as :attr:`CliResult.list_cost_usd`.
 - **Errors.** A failed call raises :class:`CliCallError` with ``retryable`` set:
   rate limits, overload, 5xx and timeouts are retried; an unknown model, a missing
-  login or a missing binary are not.
+  login or a missing binary are not. Those, and a usage limit, also set ``run_wide``.
 
 :func:`detect` reports whether a backend's binary is installed and signed in
 without calling a model (``cli.py backends``).
@@ -94,11 +94,16 @@ def _text_retryable(lowered: str) -> bool:
 
 
 class CliCallError(RuntimeError):
-    """A CLI call that produced no reply; ``retryable`` is what :func:`dokimi.llm.is_retryable` reads."""
+    """A CLI call that produced no reply; ``retryable`` is what :func:`dokimi.llm.is_retryable` reads.
 
-    def __init__(self, message: str, *, retryable: bool):
+    ``run_wide`` marks a failure every later call would meet too (no login, a usage limit, an unknown
+    model, no binary), which :func:`dokimi.novelty.routing_stop.is_run_wide` reads.
+    """
+
+    def __init__(self, message: str, *, retryable: bool, run_wide: bool = False):
         super().__init__(message)
         self.retryable = retryable
+        self.run_wide = run_wide
 
 
 @dataclass(frozen=True)
@@ -176,14 +181,14 @@ def _claude_error(payload: Dict[str, Any], done: process.Completed) -> CliCallEr
     lowered = message.lower()
     if any(w in lowered for w in ("authentication", "not logged in", "/login")):
         return CliCallError(f"claude-cli: {message} (sign in: claude setup-token, or claude /login)",
-                            retryable=False)
+                            retryable=False, run_wide=True)
     if "usage limit" in lowered:  # a subscription cap: it resets in hours, a retry cannot help
-        return CliCallError(f"claude-cli: {message}", retryable=False)
+        return CliCallError(f"claude-cli: {message}", retryable=False, run_wide=True)
     if status is None:  # no structured status: the words, and a status the CLI printed as text
         retryable = _text_retryable(lowered)
     else:  # a structured status decides: a 400 that quotes "rate_limit" is still a 400
         retryable = status in _RETRYABLE_STATUS
-    return CliCallError(f"claude-cli: {message}", retryable=retryable)
+    return CliCallError(f"claude-cli: {message}", retryable=retryable, run_wide=status == 404)
 
 
 def _main_model(model_usage: Dict[str, Any], requested: str) -> Optional[str]:
@@ -217,9 +222,9 @@ def _claude_parse(done: process.Completed, model: str, provider: str) -> CliResu
         # No JSON, so no status field: the transient words and the status printed as text are checked.
         # A subscription cap is not transient, as on the JSON path, whatever status it prints.
         lowered = text.lower()
-        retryable = "usage limit" not in lowered and _text_retryable(lowered)
+        capped = "usage limit" in lowered
         raise CliCallError(f"claude-cli: exit {done.returncode}, no JSON reply: {text[:300]}",
-                           retryable=retryable)
+                           retryable=not capped and _text_retryable(lowered), run_wide=capped)
     if payload.get("is_error") or done.returncode != 0:
         raise _claude_error(payload, done)
     u = payload.get("usage") or {}
@@ -432,7 +437,7 @@ def call(provider: str, *, model: str, prompt: str, system: Optional[str] = None
                                            env=_call_env(cli, cwd), cwd=cwd, input=prompt, capture_limit=None)
         except FileNotFoundError as exc:
             raise CliCallError(f"{provider}: '{cli.binary}' is not installed or not on PATH ({exc})",
-                               retryable=False) from exc
+                               retryable=False, run_wide=True) from exc
     return cli.parse(done, model, provider)
 
 

dokimi/dokimi/novelty/all_vs_all.py

@@ -218,6 +218,7 @@ def _model_arm(candidates: list[Candidate], queries_by_skill: dict[str, list[str
     first_error = None
     planned = done = 0
     capped: str | None = None
+    stop_reason = "spend_cap"
     for index, name in enumerate(names):
         if name not in queries_by_skill:
             continue
@@ -247,6 +248,7 @@ def _model_arm(candidates: list[Candidate], queries_by_skill: dict[str, list[str
         # A failed call measured nothing: it is not a pick, so it is left out of the counts.
         model[name] = Counter(row.get("picked") or NO_PICK for row in rows if not row.get("error"))
         capped = routed.get("cap_stop")
+        stop_reason = routed.get("stop_reason", "spend_cap")
         # Failed calls, and calls the cap refused: neither measured anything.
         unscored_here = routed["n_failed"] + (len(queries) * repeats - len(rows) if capped else 0)
         if unscored_here:
@@ -258,7 +260,7 @@ def _model_arm(candidates: list[Candidate], queries_by_skill: dict[str, list[str
                model=client.model, seed=seed, repeats=repeats, noise=noise,
                n_failed=n_failed, n_abstained=n_abstained, first_error=first_error)
     if capped is not None and done < planned:  # a stop on the last planned call leaves none unmade
-        arm["partial"] = partial_record(capped, done, planned)
+        arm["partial"] = partial_record(capped, done, planned, stop_reason)
     return arm
 
 

dokimi/dokimi/novelty/noise.py

@@ -9,9 +9,10 @@
 from collections import Counter
 
 
-def partial_record(message: str, calls_done: int, calls_planned: int) -> dict:
-    """The marker of a result cut short by ``--max-usd``: what ran, of what was planned."""
-    return {"reason": "spend_cap", "calls_done": calls_done, "calls_planned": calls_planned,
+def partial_record(message: str, calls_done: int, calls_planned: int,
+                   reason: str = "spend_cap") -> dict:
+    """The marker of a result cut short by ``--max-usd`` or a permanent error: what ran, of what was planned."""
+    return {"reason": reason, "calls_done": calls_done, "calls_planned": calls_planned,
             "message": message}
 
 

dokimi/dokimi/novelty/routing.py

@@ -48,6 +48,7 @@
 from .catalog_keep import check_keep_copies_under, keep_past_broken_copies, resolve_kept_copies
 from .client import LLMClient, OutputTruncated, SpendCapReached, parse_json_object
 from .noise import partial_record, repeat_noise
+from .routing_stop import PERMANENT_STOP, is_run_wide
 
 if TYPE_CHECKING:
     from dokimi.external.validate import SkillReview
@@ -265,11 +266,13 @@ def _pick_row(base: dict, reply: str, order: list[int], candidates: list[Candida
             "order": [candidates[i].name for i in order]}
 
 
+
 def _ask_once(client, prompt: str, base: dict, order: list[int], candidates: list[Candidate],
               gold: int) -> tuple[dict | None, str | None]:
-    """One call of the routing loop: its row, and the cap's message when the cap stops the run.
+    """One call of the routing loop: its row, and the message when the cap or a permanent error stops the run.
 
     A call the cap refused has no row; one with no price was paid for, so its reply is scored (a cut one as a decline) and the caller stops.
+    A run-wide provider error (:func:`dokimi.novelty.routing_stop.is_run_wide`) has no row either and stops the run.
     """
     try:
         reply = client.text(prompt, accept=partial(_is_pick_reply, n_cards=len(order)),
@@ -282,6 +285,9 @@ def _ask_once(client, prompt: str, base: dict, order: list[int], candidates: lis
             return {**base, "picked": None, "abstained": True, "raw_reply": exc.reply[:_RAW_REPLY_CHARS]}, stop
         return _pick_row(base, exc.reply, order, candidates, gold), stop
     except Exception as exc:  # noqa: BLE001 - one failed query must not lose the rest
+        if is_run_wide(exc):
+            # No credential, wrong model ID or missing permission fails every later call alike (#677).
+            return None, PERMANENT_STOP +(str(exc)[:200] or type(exc).__name__)
         return {**base, "picked": None, "error": str(exc)[:200] or type(exc).__name__}, None
     return _pick_row(base, reply, order, candidates, gold), None
 
@@ -339,7 +345,10 @@ def model_routing(
 
         When the client's ``--max-usd`` cap refuses a call, or a call comes back unpriced, the
         loop stops and the result holds the calls made so far, with ``cap_stop`` (the cap's
-        message). A ``partial`` block (:func:`partial_record`) is added only when planned calls
+        message). A run-wide provider error (:func:`~dokimi.novelty.routing_stop.is_run_wide`)
+        stops it the same way, with the error in ``cap_stop`` and ``stop_reason``
+        ``"permanent_error"`` (else ``"spend_cap"``); any other error is a row of its query. A
+        ``partial`` block (:func:`partial_record`) is added only when planned calls
         were left unmade (``n_calls`` below queries x repeats). A query stopped between repeats
         keeps its rows but is left out of ``noise``. Without a cap that trips, there is no
         ``cap_stop`` key.
@@ -393,8 +402,10 @@ def model_routing(
         # The stop is kept apart from ``partial``: a stop on the run's last planned call leaves
         # nothing unmade, but a caller (all-vs-all) still must not ask a later skill unmetered.
         result["cap_stop"] = capped
+        reason = "permanent_error" if capped.startswith(PERMANENT_STOP) else "spend_cap"
+        result["stop_reason"] = reason
         if len(picks) < len(queries) * repeats:
-            result["partial"] = partial_record(capped, len(picks), len(queries) * repeats)
+            result["partial"] = partial_record(capped, len(picks), len(queries) * repeats, reason)
     return result
 
 
@@ -434,43 +445,6 @@ def headings_as_queries(body: str, *, limit: int = 40) -> list[str]:
     return [h for h in headings if len(tokenize(h)) >= 2][:limit]
 
 
-# Output tokens per selection call, measured on one run (360 calls drew 23.3k tokens).
-SELECTION_REPLY_TOKENS = 65
-
-
-def selection_prompt_tokens(candidates: list[Candidate], queries: list[str]) -> list[int]:
-    """The prompt tokens (chars / 4) of each call one repeat of :func:`model_routing` makes.
-
-    One entry per sent query, in query order. :func:`model_routing` sends every query, contentless
-    ones too, unless none has a content word: then the run refuses before any call, and the list
-    is empty. Each call carries the whole field's cards in the selection prompt.
-    """
-    cards = "\n".join(f"{position}. {candidate.card}"
-                       for position, candidate in enumerate(candidates, 1))
-    sent = list(queries) if any(tokenize(query) for query in queries) else []
-    return [len(_SELECT_PROMPT.format(query=query, cards=cards)) // 4 for query in sent]
-
-
-def selection_estimate(candidates: list[Candidate], queries: list[str], *,
-                       repeats: int = 1) -> dict:
-    """The calls and tokens :func:`model_routing` would make, counted without making them.
-
-    Every repeat sends the same prompts (see :func:`selection_prompt_tokens`). Each reply is
-    billed at :data:`SELECTION_REPLY_TOKENS` output tokens, the measured figure, and
-    ``output_tokens_per_call`` says which. Zero repeats send nothing, so no query is sent.
-    """
-    prompts = selection_prompt_tokens(candidates, queries) if repeats else []
-    calls = len(prompts) * repeats
-    return {
-        "n_queries": len(queries),
-        "n_queries_sent": len(prompts),
-        "n_calls": calls,
-        "input_tokens": sum(prompts) * repeats,
-        "output_tokens_per_call": SELECTION_REPLY_TOKENS,
-        "output_tokens": calls * SELECTION_REPLY_TOKENS,
-    }
-
-
 def non_skill_reason(ref: str) -> str | None:
     """Why a ``skill:`` reference names something other than a skill, or ``None``.