[antonwolfy]

Summary

Issue #624 reports two problems. The first is that skill novelty analyze loses everything it paid for when --max-usd stops it. On main, a capped run (answers cost 0.6 USD each under --max-usd 1) exits with the measurement stopped: 11 probe answers were not run or graded; spend cap reached: $1.20 of $1.00 after 9 calls (next call: judge-m). Raise --max-usd, or measure less (a cheaper --tier, a lower --max-atoms); nothing was written, and the output directory holds nothing: no atoms, no probe answers. atoms and probe already keep a partial result at the cap; analyze did not. A cap in extraction, the faithfulness check, the need graph, probe wording, classification or attribution lost its work the same way.

Issue	On main	Now
#624 problem 1: analyze stopped by --max-usd	exit 1, nothing was written	novelty.partial.json beside novelty.json with status: "partial", the stage reached, the extracted atoms, the spans and the probe answers and judgments so far; exit 1 naming the file
#624: how to keep the paid calls	not said	the message says a rerun with a higher --max-usd and the same DOKIMI_RESPONSE_CACHE replays them at $0, or that the cache was not set on this run
#624 problem 2: --estimate-only below the real bill		not changed here, see below

Taken: a partial sidecar written with the helpers and sidecar rules atoms and probe use (<out stem>.partial.json, never over a finished file, FIFO/read-only/long-name handling shared), because the pieces already exist and the readers' refusal model is the same. The exit code stays 1.

skill shorten shares no code with this path and still writes nothing at the cap; this is tracked in #630. Problem 2 (the estimate) needs measured runs per tier and role and touches estimate_usd, which an open PR moves to novelty/cost.py; it is tracked in #631. Exit code 3 for a budget stop that kept its partial output would change atoms, probe, routing and analyze together against a documented table, and is tracked in #632. What this PR leaves of the issue is tracked in the three follow-ups, so closing the issue loses none of it.

Closes #624

Changes

• dokimi/novelty/analyze.py: analyze wraps the pipeline body (_analyze) and records each stage and what it produced; a spend stop anywhere returns an AnalyzeResult with partial set instead of raising. A cap that stopped extraction (already recorded on the extraction report, and ignored by analyze before) now ends the run there. The probe stage hands build_probe_set the empty set on Progress (into=), which the builder fills in place, so a cap that falls between the paraphrase call and the reversal call keeps the direct probes and the accepted paraphrases.
• dokimi/novelty/analyze_partial.py (new): the partial record. status: "partial" and a partial block (reason, stage, message, cause, spans_read/spans_total or calls_done/calls_planned), extracted_atoms, source_spans, the probe set once built (its counts and fidelity figures, plus every probe's wording and answer key in probes.items, so a stop before the first answer keeps what the wording calls paid for) and runs. It holds no paired, arms, k0 or view summaries, since the cut leaves the arms uneven.
• dokimi/commands/novelty_budget.py, novelty_run.py, novelty.py: analyze saves the record through write_run and exits with the message, the file and the DOKIMI_RESPONSE_CACHE hint (true for these calls: every novelty call goes through the response cache, and a cut-off or unusable reply is asked again). The hint promises the replay only when the cache directory is writable and no cache write failed during the run (caching.response_store_failed, set when store_response hits an OSError such as a full disk); otherwise it says a rerun pays again. After a no_price stop the hint says to rerun with the model priced, or without --max-usd, since a higher cap does not clear it; a spend_cap stop still says a higher --max-usd.
• dokimi/artifacts.py: load_artifact("novelty", ...) refuses a partial file by name, whether it carries status: "partial", the partial block or both, with or without validation, which covers novelty view and --novelty-artifact; save_artifact("novelty", ...) refuses such a record with an InputError and writes nothing, so the library path cannot save it as a finished run (the command writes the partial file through write_run). dokimi/novelty/items.py: a probe file read by skill novelty items is refused on either marker too. The record opens with status, so runs skips it, as it does atoms.partial.json.
• dokimi/schemas/novelty.v1.json: widened in place with status, partial, extracted_atoms, source_spans and runs; paired, k0 and spans leave required and validate_artifact requires them of every record without status (a schema allOf was reported as breaking by schema_compat). That requirement is artifacts.summary_errors, which validate_artifact and runs show both apply, so runs show reports a finished record without them as invalid.
• The partial record's atoms.n is the probed sample once the cap's sample is chosen, as a finished record counts it (max_atoms=1 on a two-atom skill: n: 1, sampling: cap-spread:1 in both), and every extracted atom for a stop in extraction, where sampling is none (documented in the schema). A stop in extraction also keeps run_config (seed, max_atoms, arms, budgets, prompts_sha256). The schema says calls_done counts the answers paid for, the ungraded ones included. analyze()'s docstring says a caller checks result.partial.
• docs/skill-novelty.md: the two sentences that said analyze loses its work now describe the partial file.
• scripts/growth_baseline.json: analyze.py::analyze is _analyze (complexity 21, unchanged).
• Changelog fragment (Fixed).

Risk

A capped analyze still exits 1 and writes no novelty.json; it now also leaves novelty.partial.json. A finished run writes what it wrote before. A script that checks the output directory for emptiness after a cap sees one more file. novelty.v1 accepts the partial record and no longer lists paired, k0 and spans as required in the schema file itself; validate_artifact and runs show enforce them for any record without status, but a tool that validates with the schema file alone would accept a finished record without them.

Testing

On a Linux review host, at b74a588 (on main fc9fbf5):

• make ci-local: 32 passed, 0 failed, 13 skipped (docker/network smoke, Claude Code install, the other Python matrix legs, conda-forge, fuzz, semgrep, test-latest and the other jobs that do not run on a PR). 19029 tests passed, 19 skipped; changed-line coverage is reported per file.
• Repro: the new tests in tests/test_novelty_analyze_cap_partial.py (a cap in extraction, the faithfulness check, the measurement and the classify stage, a call with no price, the command, the finished file untouched, readers refusing the file, runs skipping it) all fail on main (the run exits with nothing was written) and pass here.
• Mutants, each failing the new tests: dropping the write (4 fail), dropping status from the record (5 fail), making load_artifact accept a partial file (1 fails), writing over novelty.json instead of the sidecar (3 fail), and treating a DOKIMI_RESPONSE_CACHE that names a file as an active cache (the test for that branch fails).
• Mutants for the review round, each failing its test: loading a record by partial only, not status (the readers test fails), dropping probes.items from the record (the stop-after-the-wordings test fails), advising a higher cap for every stop reason (the no-price hint test fails).
• skill novelty items reads a probe file through load_probe_cells, which refused a partial file only by its partial key: a stub probe file with status: "partial" and no partial block was accepted as a finished measurement. Now refused (test_novelty_items_refuses_a_probe_file_marked_partial_by_status_alone); the test fails with the key-only check.
• A cap between the paraphrase call and the reversal call (test_a_cap_between_the_two_wording_calls_keeps_the_paraphrases_already_paid_for): the partial record keeps the direct and paraphrase probes and their fidelity row. Mutant: not sharing the Progress probe set with build_probe_set (assigning it after the build returns) leaves the record without probes, and the test fails with KeyError: 'probes'.
• A cache write that fails on a full disk (OSError ENOSPC from the write, in a directory that passes the writable check) used to leave the message promising a $0 replay; now it says a rerun pays again (test_no_replay_is_promised_when_a_cache_write_failed_on_a_full_disk). Mutant: not recording the failed directory in store_response fails it; on main it fails too.
• A capped analyze with DOKIMI_RESPONSE_CACHE set, then a rerun with a higher cap on the same cache through the real LLM path (test_a_rerun_with_a_higher_cap_and_the_same_cache_replays_the_paid_calls_at_zero): the rerun sends none of the prompts the first run paid for, spends only on the fresh ones, and keeps the same two atoms. Mutant: store_response storing nothing fails it (the rerun asks the model for the paid prompts again).
• runs show on a finished novelty.json without paired, k0 and spans lists three errors, and shows a record with status: "partial" as valid (test_a_finished_novelty_record_without_its_summaries_is_invalid_but_a_partial_one_is_not). Mutant: runs show using the schema errors alone fails it.
• A wording call with no price whose reply holds no JSON array is paid for but unreadable: the run still stops with no_price and each atom asked for is tallied as a rejected empty wording (test_a_no_price_wording_reply_that_is_not_json_keeps_no_wording_and_still_stops, paraphrase and reversal). Mutant: skipping that tally when the reply cannot be parsed fails both cases.
• Mutants for the atoms count, run config and save checks, each failing its tests: counting len(progress.extracted) for atoms.n (test_a_capped_run_stopped_in_measurement_counts_the_probed_sample_as_a_finished_one_does and test_a_cap_in_the_need_graph_counts_the_sample_not_the_extraction fail), not setting run_config before the extraction stop (test_a_cap_in_extraction_keeps_the_atoms_read_and_marks_the_rest_unread fails), and removing the partial check from save_artifact (test_save_artifact_refuses_a_partial_novelty_record_and_writes_nothing fails for status, partial and both).
• Three existing tests in tests/test_novelty_cap_partial.py that expected analyze to raise now check the partial block.
• The Linux 3.11 and 3.14 CI legs failed 25 tests (test_llm_delegates, test_judge_sampling, test_model_defaults, test_openai_compatible_route, test_llm_retry_credentials) on pydantic's ReadOnly TypedDict UserWarning raised by a newer LiteLLM. They do not touch this PR; main ignores that warning for any class since 3b61dd0 (test: ignore pydantic's ReadOnly TypedDict notice for any class #610), which this branch now contains. Those five files pass here at the merge of main (117 passed).
• Not run: Python 3.11, 3.12 and 3.14, the Windows and wheel matrix legs, and any real model call.

Revert

A plain revert restores the old behaviour (a capped analyze writes nothing). Nothing else is stored; a novelty.partial.json already written stays on disk, is no longer refused by name, and would fail the old schema, which requires paired, k0 and spans.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 18 selected item(s).

CI at b74a588: 24 passing, 2 skipped.

This PR has reached its review limit, so only a re-request starts a review.

Review details

• Commit reviewed: b74a588 against merge base fc9fbf5
• Coverage: 18 files in 4 groups
• Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
• Usage: 679k tokens (468k from cache), 55s
• Tool calls: 53 (36 file_read, 5 file_read_diff, 4 code_search, 4 pr_threads, 2 ci_status, 2 local_check_results)
• Partial novelty analysis, caching, artifacts, and schema: 9 files
• Novelty cap and budget command integration and documentation: 6 files
• Run display updates and tests: 2 files
• Growth baseline update: 1 file

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.71%. Comparing base (fc9fbf5) to head (b74a588).
⚠️ Report is 5 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #633      +/-   ##
==========================================
+ Coverage   95.67%   95.71%   +0.03%     
==========================================
  Files         394      397       +3     
  Lines       53026    53247     +221     
  Branches     9137     9161      +24     
==========================================
+ Hits        50733    50963     +230     
+ Misses       1417     1410       -7     
+ Partials      876      874       -2     

Files with missing lines	Coverage Δ
dokimi/dokimi/artifacts.py	95.41% <100.00%> (+0.50%)	⬆️
dokimi/dokimi/caching.py	98.76% <100.00%> (+0.03%)	⬆️
dokimi/dokimi/commands/novelty.py	99.76% <100.00%> (+<0.01%)	⬆️
dokimi/dokimi/commands/novelty_budget.py	96.19% <100.00%> (+0.73%)	⬆️
dokimi/dokimi/commands/novelty_run.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/analyze.py	96.83% <100.00%> (+0.40%)	⬆️
dokimi/dokimi/novelty/analyze_partial.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/atoms.py	98.79% <ø> (ø)
dokimi/dokimi/novelty/items.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/probes.py	100.00% <100.00%> (+0.62%)	⬆️
... and 1 more

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

The red Test Linux Python 3.11 and Test Linux Python 3.14 checks are not from this PR: they fail 25 tests in five litellm test files that the PR does not touch, because CI installs litellm 1.109.0 and the suite runs with warnings as errors (#609). The fix is #610. Waiting for #610 to merge, then I will merge main into this branch and recheck.

[vchamarthi]

Review at 0fc4dc2. 3 new findings: 2 must-fix, 1 nit. The change makes a --max-usd stop in skill novelty analyze keep its paid work in novelty.partial.json instead of writing nothing, and makes the readers of a finished novelty.json refuse that file. The changed tests pass on 0fc4dc2, and a loader that checks only the partial key is caught. One must-fix: a spend stop in extraction leaves cause out of its partial block, which the schema and docs say a spend stop carries.

Verified

• Finished run unchanged: A run that the cap does not stop returns no partial block and no status key, as on main.
• Cap in extraction: A cap that stops extraction after 50 of 200 spans keeps the 50 atoms, marks the 150 other spans unread, and the record validates against novelty.v1.
• Cap between the two wording calls: A cap that falls after the paraphrase call and before the reversal call keeps the direct and paraphrase entries in probes.items, and has no reversal kind.
• Call with no known price: A call with no price stops the run with reason: no_price and no cause, and the rerun advice does not mention a higher --max-usd, even with DOKIMI_RESPONSE_CACHE set.
• Cache variable names a file: When DOKIMI_RESPONSE_CACHE names a file rather than a directory, the stop message says the cache was not set on the run, and does not promise a replay.
• Readers refuse a partial novelty file: load_artifact("novelty", ...) raises for a partial file with or without validation, and novelty view, the subjects loader and datasets_export all go through it.
• Status-only partial skill novelty probe file: A skill novelty probe file that carries status: "partial" and no partial block is refused by load_probe_cells.
• Loader refusal of status-only partial: A loader that checks only the partial key lets a status-only partial record through, and the readers test fails on it.
• Merge with main: The PR merges onto current main with no conflict.

Not verified

• The replay promise in the changelog fragment, the docs and the stop message (a rerun with a higher --max-usd and the same DOKIMI_RESPONSE_CACHE replays the paid calls at $0) is not tested end to end. No test runs a capped analyze with the cache set, then a second run on the same cache, and checks the second run's spend and atoms: Add that two-run test: cap analyze with DOKIMI_RESPONSE_CACHE set, rerun with a higher cap on the same cache, and assert zero spend on the replayed calls and the same atoms.

Pre-existing, not blocking

• dokimi/dokimi/novelty/atoms.py:787: load_atoms_record refuses an atoms file only when it has a partial key. An atoms file with "status": "partial" and no partial block loads as a finished atom set: on 0fc4dc2 and on main, load_atoms_record returns the atoms and raises nothing for such a file. The PR fixes the same check for novelty files (dokimi/dokimi/artifacts.py:321) and not this one; the thread names it as tracked in #642.

Findings outside the diff

dokimi/dokimi/schemas/novelty.v1.json:13

nit: The published novelty.v1 schema no longer requires paired, k0 or spans, so a schema-only check of a finished record passes without them.

On 0fc4dc2 the required list at lines 7-13 names five keys; on main it names eight, with paired, k0 and spans. dokimi/runs/show.py:147 checks a record with schema_errors(schema, data) alone, so a finished novelty.json without the three summaries gets no error there; on main the same check reports the missing key. validate_artifact (dokimi/artifacts.py:215) still enforces them on save and load, so save_artifact and load_artifact are unchanged. The PR's risk note names this for tools that validate with the schema file, and runs show is one of them.

Also at: dokimi/dokimi/runs/show.py:147.

Suggestion: Have runs show also run the summary check of validate_artifact for a record without status, or move that check into schema_errors for novelty.v1 so both readers share it.

dokimi/dokimi/novelty/probes.py:471

must-fix: A no-price stop in the wording stage discards the paraphrase wordings its paid reply carried, so probes.items keeps only the direct entries.

On 0fc4dc2, analyze run with SpendCap(limit_usd=1.0) through the _run helper of dokimi/tests/test_novelty_analyze_cap_partial.py with cost_of=lambda prompt: None if 'Reword each question' in prompt else 0.0 returns a partial record with reason: no_price at stage: probes. Its probes.items holds only direct entries, although the scripted transport returned valid paraphrases. The reply is on SpendCapReached.reply (dokimi/dokimi/novelty/client.py:183), but _generate catches only Exception at line 471, and SpendCapReached is a BaseException, so the exception leaves build_probe_set before any paraphrase is appended. On main the same input raises SpendCapReached out of analyze and writes no record, so main has no partial record to compare. The changelog fragment says the partial record keeps the generated wordings in probes.items, and it does not limit that to the spend cap. No test covers this: test_a_call_with_no_price_stops_analyze_and_is_named_as_such asserts reason, the absence of cause and the message, and test_a_cap_between_the_two_wording_calls_keeps_the_paraphrases_already_paid_for uses a spend cap, not a no-price stop.

Suggestion: Read the wordings from SpendCapReached.reply in _generate when it is set, as dokimi/dokimi/novelty/atoms.py:309-313 does for extraction, and raise the stop only after build_probe_set has appended them to into; the reversal call needs the same handling. Assert the paraphrase kind in the no-price test.

[xaleryb]

• Ran: this branch with current main merged in, full suite, ruff, mypy, bandit -ll, the growth, comment-block and changelog gates, and docs check --strict, on Python 3.13. The suite passes (18998; one test_process timeout under load, green alone three times) and every gate is clean.
• Pushed 872d83c:
  • Merge of main. commands/novelty.py conflicted on the novelty_run import line: refactor(novelty): role clients, spend cap and USD estimate move to novelty/cost.py #625 dropped spend_cap from it, this PR adds exit_if_analyze_partial. With that line resolved, 7 command tests in tests/test_novelty_analyze_cap_partial.py -> AttributeError: ... no attribute '_clients' -> expected green. refactor(novelty): role clients, spend cap and USD estimate move to novelty/cost.py #625 moved the clients, so they now patch dokimi.novelty.cost.clients, and the unused novelty_cmd import is gone.
  • 065f019: a no-price stop in wording generation -> the paraphrase and reversal wordings its paid reply carried were lost -> expected kept in the partial probe set. They are now kept.
  • 9f6f638: an extraction --max-usd stop -> partial block with no cause -> expected cause, as novelty.v1 says. It now has one.
  • 821ed1c: DOKIMI_RESPONSE_CACHE under a regular file or in a read-only directory -> nothing stored, yet the message promised a $0 replay -> expected no promise. It no longer makes one.
  • b4f3944: tests for the partial record's cost, run_config, sampling, faithfulness and the need_graph stage.
• Left for the author: nothing blocking; the novelty.v1 nit (paired/k0/spans no longer required) is still open.

[antonwolfy]

(commented) Review at 0fc4dc2. 3 new findings: 2 must-fix, 1 nit. The change makes a --max-usd stop in

@vchamarthi (#633 (review)): No-price wordings (must-fix): already kept, since 065f019 (test_a_no_price_wording_call_keeps_the_wordings_its_reply_carried). Two-run replay test: added in 56e0570 (test_a_rerun_with_a_higher_cap_and_the_same_cache_replays_the_paid_calls_at_zero): a capped analyze with the cache set, then a rerun with a higher cap through the real LLM path; the rerun sends none of the prompts already paid for, spends only on the fresh ones, and keeps the same atoms. runs show nit: summary_errors in artifacts.py now holds the paired/k0/spans check that validate_artifact and runs show both apply, so a finished record without them is invalid in runs show and a partial one is not (aefc7e2). The status-only partial atoms file at atoms.py:787 is outside this PR and tracked in #642.

[antonwolfy]

• Ran: this branch with current main merged in, full suite, ruff, mypy, bandit -ll, the growth, comment-block and changelog gates, and docs check --strict, on Python 3.13. The suite passes (18998; o

@xaleryb (#633 (comment)): A status comment on commits already in the head. The novelty.v1 nit it leaves open is the runs show check fixed in aefc7e2.

[vlad-perevezentsev]

Review at aefc7e2. The change is right: a --max-usd or no-price stop at any stage of analyze keeps what was paid for in <out stem>.partial.json, and every reader of a finished novelty.json refuses it. What is left is small: a few fields of the partial record mean something else than in a finished one, and a library path that can still save it as finished. Details are in the line comments.

What I ran, at aefc7e2:

• make ci-local --offline: 0 failed, 21 passed, 19 skipped (network-only jobs); the suite 18,968 passed, 72 skipped.
• A capped analyze with max_atoms=1 (measurement stop vs. finished run), a save_artifact of the partial record followed by runs.scan.head_version, and an extraction stop with seed=7, max_atoms=30 to see its run_config, sampling and warnings.

Checked, no issues

• Earlier threads: the status-only partial refusal (load_artifact, load_probe_cells), the reason-aware replay hint, the probe wordings kept inside and between the wording calls and from a no-price reply, cause on an extraction stop, the full-disk cache case and the runs show summary check are all in the code and each has a test.
• Stage tracking: SpendCapReached is a BaseException, so the except Exception blocks in the wording and need-graph code do not swallow it; each stage sets progress.stage before its first paid call.
• Schema and docs: novelty.v1 is only widened (fewer required keys, new optional status, partial, probes.items), which schema_compat accepts; the docs and the changelog fragment match the behaviour; the new files have the SPDX header.

[antonwolfy]

(commented) Review at aefc7e2. The change is right: a --max-usd or no-price stop at any stage of `analyze

@vlad-perevezentsev (#633 (review)): The review's points are its four threads above (atoms.n, the extraction stop's run_config and sampling, save_artifact, the calls_done wording), settled in 3038f94 and b74a588.

[vlad-perevezentsev]

LGTM

I checked b74a588. 3038f94 settles all four points from my last round: atoms.n counts the probed sample as a finished record does, an extraction stop keeps run_config and sampling: none is documented, save_artifact refuses a partial novelty record, and the calls_done wording matches the code. Leaving the warnings of an extraction stop empty is fine with me: unread_span_ids and spans_read/spans_total already say what was not read.

Merge: safe now: it merges cleanly into current main, 5 commits behind, and its tests pass on the merge. Once it is in, #702 (docs/skill-novelty.md), #595 (commands/novelty.py) and #472 (novelty/probes.py) conflict with it and need a rebase.

What I ran, at b74a588:

• make ci-local --offline: 0 failed, 21 passed, 19 skipped (network-only jobs); the suite 18,976 passed, 72 skipped.
• On this branch merged with current main: the analyze cap, probe cap, runs show and analyze tests, 201 passed.

Checked, no issues

• New tests: the capped measurement and need-graph stops assert atoms.n == 1 with two atoms extracted, the extraction stop asserts the seed and max_atoms in run_config, and the save_artifact test covers either marker alone; each asserts a value the code before 3038f94 got wrong.
• Docs and changelog: the fragment and skill-novelty.md describe the refusal in save_artifact, the atoms.n rule and the library caller's result.partial check.

[antonwolfy]

(approved) LGTM

@vlad-perevezentsev (#633 (review)): Approval with no change requested. I checked the refusal you describe at dokimi/dokimi/artifacts.py:277 (save_artifact raises InputError for a novelty payload with status or partial, via _is_partial_novelty at line 326). No new commit this round.