[antonwolfy]

Summary

#708: the per-item except Exception in atom extraction, the faithfulness check, probe wording generation, probe answers and grades, the batched value and conflict calls and the retention adjudication had no run-wide check, so with missing credentials each made, and recorded as failed, every remaining call (the same cause #709 fixed in novelty routing). On main, with a client whose every call raises an authentication error ("Unable to locate credentials"):

Step	main	Here
extract_atoms, 200 spans in 8 batches	8 calls, then ExtractionError: no atoms: every extraction call failed	1 call; report.partial has reason permanent_error, the spans are marked unread
run_measurement, 10 probes in 2 arms, 1 worker	20 answer calls, 20 runs each an ERRORED answer, not_run 0, stop None	1 answer call, 0 runs, not_run 20, stop ("permanent_error", "permanent error: Unable to locate credentials")
ask_in_batches (value and conflict batches), verify_atoms, probe wordings, the retention oracle	a call per batch, atom, chunk or candidate, each recorded as failed	the first call stops the step

The fix takes is_run_wide from routing_stop.py, as #709 does, and not is_retryable alone (one of the issue's two options). A context-length rejection, a refusal or an odd reply is non-retryable and yet one item's own failure, so stopping on it would lose the rest of a run; each site has a test that it does not stop. A run-wide error is raised as the SpendCapReached the cap stops use, with reason permanent_error, so every path that keeps a cap stop's partial result keeps this one's.

Closes #708

The same cause in a site the issue does not name was searched for in dokimi/novelty/ and the novelty commands; the sites found are all fixed here (see Changes), so there is no follow-up.

Changes

• novelty/routing_stop.py: permanent_stop(exc) builds the stop (reason permanent_error, message permanent error: <error>) and stop_if_run_wide(exc) raises it when is_run_wide(exc).
• novelty/atoms.py: extract_atoms and verify_atoms call the model through batching.or_stop, which raises that stop for a run-wide error. In extraction it lands in the existing except SpendCapReached, so the atoms read stay and the unread spans are marked, as at the cap. A partial atoms file with this reason is refused with "Fix the error and rerun", not a --max-usd hint.
• novelty/measure.py: run_measurement and _grade raise the stop first in their except Exception handlers. Jobs not yet started when it is recorded are not run (not_run); an answer already paid for whose grade met the error stays, ungraded, as for a cap. At most concurrency calls are in flight when it is recorded.
• novelty/routing_stop.py: is_run_wide is false for a ValueError, which is what the JSON extractors raise for a reply that will not parse and quote the reply in; a model that answers with "Unable to locate credentials" is one failed item, not a provider error. Provider errors are not ValueErrors.
• novelty/measure.py: with concurrency above 1, a worker whose answer returns after another recorded the stop keeps it ungraded (_grade_unless_stopped, through the capped-grade path) and sends the judge nothing.
• novelty/batching.py: ask_in_batches stops at a run-wide error, returns the chunks answered so far, puts the chunk and those not yet asked in failed, and sets BatchResult.stop. classify.judge_value and classify.detect_conflicts raise it, as a cap stop propagates.
• novelty/probes.py (_generate) and novelty/retention.py (_adjudicate): the same first-statement check.
• artifacts.load_artifact, commands/novelty_budget.replay_hint and commands/novelty_atoms_file._verdicts_hint no longer advise a higher --max-usd after this stop; novelty.v1.json lists permanent_error in partial.reason.
• docs/skill-novelty.md: one sentence extends the routing passage to these steps, and the analyze partial block lists the new reason.
• tests/test_novelty_permanent_stop.py: a test per site that the step stops at the first run-wide error and keeps its partial result, and a counterexample per site that a context-length rejection of one item leaves the rest of the run going. test_a_recovered_parse_error_does_not_hide_the_failure_that_lost_spans in tests/test_novelty_atoms.py used an AuthenticationError as its example of a lost batch, which is now a stop; it uses a BadRequestError and asserts the same.
• Changelog fragment (Fixed).

Risk

A run whose extractor, answerer, judge or adjudicator has no credentials, no permission or an unknown model now ends at the first call, with a partial result and exit 1 as at the cap, where atoms raised ExtractionError and probe and analyze finished with every answer ERRORED. A partial.reason of permanent_error is new in atoms and novelty.json partial files (the schema lists it); readers that refuse partial files refuse these too. Any other failure of one item is recorded as before.

The branch merges main, which changed the routing paragraph of docs/skill-novelty.md that this change extends (the AWS credential cases) and renamed the credential marker tuple routing_stop.py reads (now _NON_RETRYABLE_MESSAGES); both conflicts are resolved keeping both sides. No other open PR touches the lines changed here (#692 changes retention.py in verdict, not _adjudicate; #729 only adds comments to the except Exception lines; #697 changes save_atoms; #489 wraps _grade per judge in run_measurement).

Testing

On a Linux review host, at ee3e586 (on main cc2ca31):

• make ci-local: 33 passed, 0 failed, 14 skipped. The skips are steps that need Docker and the network, the pinned Claude Code install, the jobs that run only on main or from another job's matrix, conda-forge, the fuzz run and semgrep. 20473 tests passed, 19 skipped.
• The repro above, against a client that always raises an authentication error: fails on main (8 and 20 calls), passes here (1 call each). The new test file against main's code: 13 of its 21 tests fail, and the 8 that pass are the counterexamples that a per-item error does not stop.
• Mutants, one break per copy, each fails the tests named in the brackets: extraction, verify_atoms, the answer, the grade and probe wording calls without the check, ask_in_batches without the check, judge_value or detect_conflicts without raise result.stop, the retention check removed, later jobs still run after the stop, the check widened to every non-retryable error in stop_if_run_wide and in ask_in_batches (the counterexamples fail), the stop message without its prefix, and the --max-usd advice kept in the atoms reader, the artifact reader and replay_hint.
• A reply that quotes a credential message (Unable to locate credentials: see the AWS docs.) as a batch reply or a judge reply: with the ValueError rule removed, the batch run stops with permanent_error and the grade run stops; here the batch is answered and the 8 grades are ERRORED with stop None. A two-worker measurement whose second answer is held until the first worker's grade has raised the authentication error: 1 judge call, 2 runs kept ungraded, 6 not run; with the check removed the second answer is graded (the test fails).
• Not run: the matrix legs on Python 3.11, 3.12 and 3.14, and CI's merge commit (the run is on the working tree).

Revert

A plain revert restores the old per-item handling. Partial files written with reason permanent_error stay valid JSON, but the restored schema would reject that reason in a novelty.json partial record.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

CI at f0e07a6: 26 passing, 3 skipped.

Review details

• Commit reviewed: f0e07a6 against merge base ee3e586
• Coverage: 1 files in 1 groups
• Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
• Usage: 73k tokens (46k from cache), 19s
• Tool calls: 8 (7 file_read, 1 code_search)
• dokimi/tests/test_novelty_permanent_stop.py: 1 file

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.72%. Comparing base (cc2ca31) to head (f0e07a6).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #767   +/-   ##
=======================================
  Coverage   95.72%   95.72%           
=======================================
  Files         415      415           
  Lines       55362    55408   +46     
  Branches     9516     9522    +6     
=======================================
+ Hits        52993    53041   +48     
+ Misses       1466     1465    -1     
+ Partials      903      902    -1     

Files with missing lines	Coverage Δ
dokimi/dokimi/artifacts.py	95.52% <ø> (ø)
dokimi/dokimi/commands/novelty_atoms_file.py	100.00% <100.00%> (ø)
dokimi/dokimi/commands/novelty_budget.py	96.26% <ø> (ø)
dokimi/dokimi/novelty/atoms.py	98.80% <100.00%> (ø)
dokimi/dokimi/novelty/batching.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/classify.py	97.40% <100.00%> (+0.05%)	⬆️
dokimi/dokimi/novelty/measure.py	98.78% <100.00%> (+0.05%)	⬆️
dokimi/dokimi/novelty/probes.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/retention.py	95.00% <100.00%> (+0.08%)	⬆️
dokimi/dokimi/novelty/routing_stop.py	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vchamarthi]

Pushed 1 commit to fix/novelty-permanent-stop (ee3e586..f0e07a6).

• Both tests now read the result: report.unchecked and fidelity["paraphrase"]["rejected"].
• The conflict-check test passes a ClassifyReport and asserts the rejected batch's atom ids are in report.unchecked.
• The wording test replies with valid paraphrases and asserts generated 3, kept 2, rejected 1 with reason empty, and that the probes kept are the two atoms whose request succeeded.
  Run: the full suite with main merged in, 20510 passed.