fix(novelty): atoms, probes, grades and batches stop at a run-wide provider error

dokimi/changelog.d/20261114_130000_novelty-permanent-stop.md

@@ -0,0 +1,3 @@
+### Fixed
+
+- `skill novelty atoms`, `probe` and `analyze` and `skill shorten` stop at the first authentication, permission, unknown-model, unsupported-parameter or no-credentials error (extraction, the faithfulness check, probe wordings, answers and grades, the value and conflict batches, retention adjudication) instead of failing every remaining call and recording each one as failed; the partial result is kept as at the cap, with `reason` `permanent_error`. A refusal, an unparseable reply (also one that quotes a no-credentials message) or a request that is too long is still one failed item. With `--concurrency` above 1, an answer that returns after another worker met the error is kept ungraded, not sent to the judge. (#708)

dokimi/docs/skill-novelty.md

@@ -838,7 +838,7 @@ so count the queries first, or pass `--max-usd` to stop the run at a spend limit
 A stored response in `DOKIMI_RESPONSE_CACHE` is replayed without a call, so a rerun can make
 fewer calls than the printed count.
 
-**A permanent provider error that every call would meet stops a routing run the way the cap does.** An authentication or permission error, an unknown model ID (`not found`), a parameter the provider does not take (every call sends the same ones) or missing credentials (with `claude-cli`: not logged in, a usage limit, no `claude` binary; with AWS: "Unable to locate credentials", a missing config profile or an expired SSO session, whichever class litellm wraps it in) would fail every later call alike, so the run stops at that call, keeps the rows made before it and writes the partial result below, with `"reason": "permanent_error"` and the error text as the message (`--max-usd` is not involved; the exit line gives `calls_done` of `calls_planned`). Any other failure is one failed row and the run goes on: a refusal, a request one query makes too long, an unparseable reply, an error the client does not recognise, a transient error that ran out of retries.
+**A permanent provider error that every call would meet stops a routing run the way the cap does.** An authentication or permission error, an unknown model ID (`not found`), a parameter the provider does not take (every call sends the same ones) or missing credentials (with `claude-cli`: not logged in, a usage limit, no `claude` binary; with AWS: "Unable to locate credentials", a missing config profile or an expired SSO session, whichever class litellm wraps it in) would fail every later call alike, so the run stops at that call, keeps the rows made before it and writes the partial result below, with `"reason": "permanent_error"` and the error text as the message (`--max-usd` is not involved; the exit line gives `calls_done` of `calls_planned`). Any other failure is one failed row and the run goes on: a refusal, a request one query makes too long, an unparseable reply, an error the client does not recognise, a transient error that ran out of retries. `skill novelty atoms`, `probe` and `analyze` and `skill shorten` stop the same way at the first such error in an atom extraction call, the faithfulness check, a probe wording, an answer, a grade, a value or conflict batch or a retention adjudication, and keep what they paid for as they do at the cap, with `"reason": "permanent_error"`; a refusal, an unparseable reply or a request one item makes too long is still one failed item.
 
 **A routing run stopped by `--max-usd` keeps what it paid for.** `skill routing --model`, single
 artifact or `catalog routing`, writes the result of the calls made before the cap (or a call with no
@@ -1151,8 +1151,8 @@ the cap stops the run).
 A partial file is for inspection; rerun to get a finished one.
 `analyze` does the same beside its `novelty.json` (`<out stem>.partial.json`, never over a
 finished file; the `--out` file, `novelty.json` in `--out-dir` by default). Its partial file has
-`"status": "partial"` and a `partial` block: `reason` (`spend_cap`, or `no_price` for a call with
-no known price), `stage` (`extraction`, `verify`, `need_graph`, `probes`, `measurement`, `classify`
+`"status": "partial"` and a `partial` block: `reason` (`spend_cap`, `no_price` for a call with
+no known price, or `permanent_error`), `stage` (`extraction`, `verify`, `need_graph`, `probes`, `measurement`, `classify`
 or `attribution`: the stage that was running), `message`, `cause` (the cap's own text, with what was
 spent), and `spans_read` / `spans_total` for an extraction stop or `calls_done` / `calls_planned`
 (probe answers) once the measurement started. It keeps the `extracted_atoms` and `source_spans`

dokimi/dokimi/artifacts.py

@@ -356,6 +356,7 @@ def load_artifact(kind: str, path: Path | str, *, validate: bool = True) -> dict
         raise InputError(f"{path} is a partial novelty artifact: the run was stopped before it "
                          "finished, so it is not a finished analysis. " + (
                              "Drop --max-usd to run unmetered" if reason == "no_price" else
+                             "Fix the provider error and rerun" if reason == "permanent_error" else
                              "Rerun with a higher --max-usd, or without the cap"))
     if validate:
         try:

dokimi/dokimi/commands/novelty_atoms_file.py

@@ -148,7 +148,8 @@ def _verdicts_hint(oracle, cache_dir, stop) -> str:
 
     stored = max(0, oracle.escalations - oracle.errors - 1)
     dropped = stop.reply is not None
-    rerun = "no --max-usd" if stop.reason == "no_price" else "a higher --max-usd"
+    rerun = ("no --max-usd" if stop.reason == "no_price" else
+             "the error fixed" if stop.reason == "permanent_error" else "a higher --max-usd")
     tail = "; the call with no price was paid for and its verdict was not stored" if dropped else ""
     if cache_dir and stored:
         return (f"the retention verdicts adjudicated before the stop ({stored}) are stored in "

dokimi/dokimi/commands/novelty_budget.py

@@ -70,6 +70,7 @@ def replay_hint(reason: str = "spend_cap") -> str:
 
     if response_cache_active():
         rerun = ("a higher --max-usd" if reason == "spend_cap"
+                 else "the error fixed," if reason == "permanent_error"
                  else "the model priced, or without --max-usd,")
         return (f". Rerun with {rerun} and the same {RESPONSE_CACHE_ENV}: the calls "
                 "already paid for replay at $0 (a cut-off or unusable reply is asked again)")

dokimi/dokimi/novelty/atoms.py

@@ -29,7 +29,7 @@
 
 from dokimi.utils import InputError, read_json_input
 
-from .batching import retry_wording
+from .batching import or_stop, retry_wording
 from .client import LLMClient, OutputTruncated, SpendCapReached, accepting
 from .spans import Span
 
@@ -419,7 +419,7 @@ def parse(reply: str) -> list:
         unparseable = False
         try:
             report.calls += 1
-            reply = client.text(prompt, system=_EXTRACT_SYSTEM, accept=usable)
+            reply = or_stop(client.text, prompt, system=_EXTRACT_SYSTEM, accept=usable)  # a run-wide error stops (#708)
             if not reply.strip():
                 # Its own cause, not a failed call: the fix is a higher output ceiling.
                 lost = ValueError(
@@ -571,8 +571,8 @@ def verify_atoms(
             continue
         source = "\n".join(by_id[s].text for s in atom.span_ids if s in by_id)
         try:
-            verdict = str(client.json_object(
-                _VERIFY_PROMPT.format(statement=atom.statement, source=source),
+            verdict = str(or_stop(
+                client.json_object, _VERIFY_PROMPT.format(statement=atom.statement, source=source),
                 usable=_names_a_faithfulness_verdict,
             ).get("verdict", "")).strip()
         except Exception:
@@ -786,9 +786,9 @@ def load_atoms_record(path) -> tuple[list[Atom], list[Span], dict]:
     if isinstance(data, dict) and ("partial" in data or data.get("status") == "partial"):
         if not isinstance(partial := data.get("partial"), dict) or not partial:  # no run writes this
             raise InputError(f"{path} is marked partial with no `partial` block: truncated or hand-edited")
-        if partial.get("reason") == "no_price":  # a call with no price: a higher cap will not help
-            raise InputError(f"{path} is a partial atoms file: {partial.get('message', '')}, so "
-                             "it is not a finished atom set. Drop --max-usd to run unmetered")
+        if (reason := partial.get("reason")) in ("no_price", "permanent_error"):  # a higher cap will not help
+            raise InputError(f"{path} is a partial atoms file: {partial.get('message', '')}, so it is not a finished atom set. "
+                             + ("Drop --max-usd to run unmetered" if reason == "no_price" else "Fix the error and rerun"))
         raise InputError(f"{path} is a partial atoms file: --max-usd stopped the run before it "
                          "finished, so it is not a finished atom set. Rerun with a higher "
                          "--max-usd, or without the cap")

dokimi/dokimi/novelty/batching.py

@@ -21,7 +21,8 @@
 from functools import partial
 from typing import Any, Generic, TypeVar
 
-from .client import LLMClient, OutputTruncated, accepting
+from .client import LLMClient, OutputTruncated, SpendCapReached, accepting
+from .routing_stop import is_run_wide, permanent_stop, stop_if_run_wide
 
 T = TypeVar("T")
 
@@ -46,6 +47,18 @@ def retry_wording(retried: int, unusable: int) -> tuple[str, str]:
     return label, ("; a higher --max-tokens avoids the extra calls" if cut else "")
 
 
+def or_stop(call: Callable[..., T], *args: Any, **kwargs: Any) -> T:
+    """``call(*args, **kwargs)``; a run-wide provider error raises its :func:`permanent_stop` instead.
+
+    Any other error passes through unchanged, for the caller's per-item handling.
+    """
+    try:
+        return call(*args, **kwargs)
+    except Exception as exc:
+        stop_if_run_wide(exc)
+        raise
+
+
 @dataclass
 class BatchResult(Generic[T]):
     """What came back, and what did not.
@@ -68,6 +81,10 @@ class BatchResult(Generic[T]):
             an unusable reply is counted in ``unusable`` instead. Errors that a
             split recovered from are not recorded, so this never blames a cause that
             lost nothing.
+        stop: Set when a run-wide provider error (:func:`~dokimi.novelty.routing_stop.is_run_wide`)
+            ended the run at its first call that met it: ``reason`` ``permanent_error``. The
+            chunks answered before it stay in ``answered``; the chunk that met it and those not
+            yet asked are in ``failed``, with no ``last_error``. A caller raises it, as a cap stop is.
     """
 
     answered: list[tuple[list[T], Any]] = field(default_factory=list)
@@ -78,6 +95,7 @@ class BatchResult(Generic[T]):
     unusable_retries: int = 0
     calls: int = 0
     last_error: str = ""
+    stop: SpendCapReached | None = None
 
 
 def ask_in_batches(
@@ -128,6 +146,10 @@ def read(chunk: list[T], reply: str) -> Any:
             replied = True
             result.answered.append((chunk, read(chunk, reply)))
         except Exception as exc:
+            if is_run_wide(exc):  # every later call meets it too: stop, the rest is unanswered
+                result.stop = permanent_stop(exc)
+                result.failed.extend(item for rest in [chunk, *pending] for item in rest)
+                break
             half = len(chunk) // 2
             # A reply cut at the ceiling is re-asked smaller, like one that will not parse.
             if (replied or isinstance(exc, OutputTruncated)) and half >= max(min_size, 1):

dokimi/dokimi/novelty/classify.py

@@ -283,6 +283,8 @@ def judge_value(
         prompt=lambda chunk: _VALUE_PROMPT.format(task=task, statements=_numbered(chunk)),
         parse=extract_json_array, check=_judge_check,
     )
+    if result.stop:  # a run-wide provider error ends the classification as a cap stop does (#708)
+        raise result.stop
     report.retried_batches += result.retried_batches
     report.unusable_retries += result.unusable_retries
     report.calls += result.calls
@@ -364,6 +366,8 @@ def detect_conflicts(
         prompt=lambda chunk: _CONFLICT_PROMPT.format(statements=_numbered(chunk)),
         parse=_conflict_rows, min_size=2, check=_conflict_check,
     )
+    if result.stop:
+        raise result.stop
     report.retried_batches += result.retried_batches
     report.unusable_retries += result.unusable_retries
     report.calls += result.calls

dokimi/dokimi/novelty/measure.py

@@ -24,15 +24,18 @@
 from __future__ import annotations
 
 import random
+from collections.abc import Callable
 from concurrent.futures import ThreadPoolExecutor
 from dataclasses import dataclass, field
+from typing import Any
 
 from dokimi.utils import InputError
 from .atoms import Atom
 from .client import LLMClient, SpendCapReached, parse_json_object, same_client
 from .controls import (REPEAT_ARM, clustered_ci, control_fields, reversal_by_arm,
                        reversal_paired)
 from .probes import Probe
+from .routing_stop import PERMANENT_REASON, stop_if_run_wide
 
 # Judge verdict scale. Deliberately finer than the collapses below so a rater
 # disagreement is visible as a confusion rather than hidden inside a boolean.
@@ -378,12 +381,13 @@ def one(job: tuple[Probe, str]) -> ProbeRun:
         except SpendCapReached as exc:  # a call with no price: its paid reply is kept, ungraded
             return _unpriced_answer_run(probe, arm, exc, stops, ungraded)
         except Exception as exc:  # a provider failure is a missing datum, not a wrong answer
+            stop_if_run_wide(exc)  # but one every later call meets stops the measurement (#708)
             response, confidence = f"<error: {exc}>", None
             verdict = ERRORED
             answered = False
         else:
             try:
-                verdict = _grade(probe, response, judge)
+                verdict = _grade_unless_stopped(stops, probe, response, judge)
             except SpendCapReached as exc:
                 return _capped_grade_run(probe, arm, response, confidence, exc, stops, ungraded)
         run = ProbeRun(
@@ -403,14 +407,36 @@ def guarded(job: tuple[Probe, str]) -> ProbeRun | None:
             return None
 
     with ThreadPoolExecutor(max_workers=max(1, concurrency)) as pool:
-        answered = list(pool.map(guarded, jobs))
+        answered = list(pool.map(_not_after_permanent_stop(stops, guarded), jobs))
     runs = [run for run in answered if run is not None]
     stop = (stops[0].reason, str(stops[0])) if stops else None
     return MeasureResult(runs=runs, arms=tuple(arms), with_arm="full",
                          not_run=len(jobs) - len(runs), ungraded=len(ungraded), stop=stop,
                          probes=list(probes))
 
 
+def _held_permanent_stop(stops: list[SpendCapReached]) -> SpendCapReached | None:
+    """The first run-wide provider error among ``stops``, or None (#708)."""
+    return next((held for held in stops if held.reason == PERMANENT_REASON), None)
+
+
+def _grade_unless_stopped(stops: list[SpendCapReached], probe: Probe, response: str, judge: LLMClient) -> str:
+    """:func:`_grade`, unless another worker met a run-wide error while this answer was in flight:
+    then that stop is raised and the caller keeps the answer ungraded, with no more judge calls (#708)."""
+    if (held := _held_permanent_stop(stops)) is not None:
+        raise held
+    return _grade(probe, response, judge)
+
+
+def _not_after_permanent_stop(stops: list[SpendCapReached], job: Callable[[Any], Any]) -> Callable[[Any], Any]:
+    """``job``, which is not run (``None``) once a run-wide provider error is among ``stops`` (#708)."""
+
+    def run(item: Any) -> Any:
+        return None if _held_permanent_stop(stops) else job(item)
+
+    return run
+
+
 def _capped_grade_run(probe: Probe, arm: str, response: str, confidence: float | None,
                       exc: SpendCapReached, stops: list[SpendCapReached],
                       ungraded: list[ProbeRun]) -> ProbeRun:
@@ -531,6 +557,7 @@ def _grade(probe: Probe, response: str, judge: LLMClient) -> str:
     try:
         raw = judge.json_object(prompt, usable=_names_a_verdict).get("verdict")
     except Exception:
+        stop_if_run_wide()  # raises a permanent stop; any other failure is an ERRORED grade
         return ERRORED
     verdict = raw.strip() if isinstance(raw, str) else ""
     if not verdict:

dokimi/dokimi/novelty/probes.py

@@ -40,6 +40,7 @@
 from dokimi.utils import InputError
 from .atoms import Atom
 from .client import LLMClient, SpendCapReached
+from .routing_stop import stop_if_run_wide
 
 PROBE_KINDS = ("direct", "paraphrase", "reversal")
 
@@ -493,6 +494,7 @@ def _generate(client: LLMClient, template: str, lines: list[str]) -> list[str]:
     try:
         rows = client.json_array(template.format(**{key: numbered}))
     except Exception:
+        stop_if_run_wide()  # raises a permanent stop; any other failure is a chunk of rejected wordings
         rows = []
     return _aligned(rows, len(lines))
 

dokimi/dokimi/novelty/retention.py

@@ -26,6 +26,7 @@
 
 from .atoms import Atom
 from .client import LLMClient
+from .routing_stop import stop_if_run_wide
 
 RETAINED = "retained"
 ABSENT = "absent"
@@ -170,6 +171,7 @@ def _adjudicate(self, atom: Atom, text: str) -> str | None:
                 accept=lambda raw: parse_verdict(raw) is not None,
             )
         except Exception:
+            stop_if_run_wide()  # raises a permanent stop; any other failure is an unusable verdict
             return None
         return parse_verdict(reply)
 

dokimi/dokimi/novelty/routing_stop.py

@@ -1,11 +1,16 @@
 # Copyright (C) 2026 Intel Corporation
 # SPDX-License-Identifier: Apache-2.0
-"""Which provider errors stop a routing run (#677); split from ``routing.py``, which stays under its size cap."""
+"""Which provider errors stop a run (#677, #708); split from ``routing.py``, which stays under its size cap."""
 from __future__ import annotations
 
+import sys
+
 from dokimi.llm import _NON_RETRYABLE_MESSAGES, is_retryable
 
+from .client import SpendCapReached
+
 PERMANENT_STOP = "permanent error: "
+PERMANENT_REASON = "permanent_error"
 # Errors that fail every later call alike: no key, no permission, an unknown model, a parameter every
 # call sends. Any other error (a context-length rejection, a refusal, an odd reply) may be one query's.
 _RUN_WIDE_ERRORS = frozenset({"authenticationerror", "permissiondeniederror", "notfounderror",
@@ -16,7 +21,25 @@ def is_run_wide(exc: Exception) -> bool:
     """Whether *exc* is a permanent provider error that every later call would meet too."""
     if is_retryable(exc):
         return False
+    if isinstance(exc, ValueError):  # a reply that would not parse quotes the reply, which may name a credential
+        return False
     if isinstance(run_wide := getattr(exc, "run_wide", None), bool):  # set by an error that knows (CliCallError)
         return run_wide
     return type(exc).__name__.lower() in _RUN_WIDE_ERRORS or any(
         marker in str(exc).lower() for marker in _NON_RETRYABLE_MESSAGES)
+
+
+def permanent_stop(exc: Exception) -> SpendCapReached:
+    """The stop a run-wide error makes, in the form a cap stop has: ``reason`` ``permanent_error``."""
+    return SpendCapReached(PERMANENT_STOP + (str(exc)[:200] or type(exc).__name__), reason=PERMANENT_REASON)
+
+
+def stop_if_run_wide(exc: BaseException | None = None) -> None:
+    """Raise :func:`permanent_stop` when *exc* (default: the exception being handled) is run-wide.
+
+    Called first in the handler of a per-item ``except Exception``, so that the one failure every
+    later call would meet ends the run as a cap stop does, and any other failure stays one item's.
+    """
+    exc = exc if exc is not None else sys.exc_info()[1]
+    if isinstance(exc, Exception) and is_run_wide(exc):
+        raise permanent_stop(exc) from exc