fix(novelty): analyze keeps what a --max-usd stop paid for in a partial file

dokimi/changelog.d/20261112_120000_analyze-cap-partial.md

@@ -0,0 +1,3 @@
+### Fixed
+
+- **`skill novelty analyze` keeps what a `--max-usd` stop paid for** — a run stopped by the cap (or by a call with no known price) used to exit with `nothing was written`. It now writes `<out stem>.partial.json` beside `novelty.json` (never over a finished file) with `status: "partial"`, the stage it reached, the extracted atoms, the spans and the probe answers and judgments so far, and exits non-zero naming the file and how a rerun with `DOKIMI_RESPONSE_CACHE` replays the paid calls at $0. `novelty view`, `--novelty-artifact` and every other reader of a `novelty.json` refuse a partial file; the `novelty.v1` schema accepts it. A record with `status: "partial"` and no `partial` block is refused too, by the `novelty.json` readers and by `novelty items` on a probe file, a stop on a call with no known price tells the user to price the model or drop `--max-usd` (a higher cap does not help it), and the partial record keeps the generated probe wordings in `probes.items`, also when the cap falls between the paraphrase call and the reversal call. The replay promise is withheld when the cache directory is not writable or a cache write failed during the run (a full disk), and `dokimi runs show` reports a finished `novelty.json` that lacks `paired`, `k0` or `spans` as invalid, as the loaders do. The partial record counts `atoms.n` as a finished one does (the probed sample), a stop in extraction keeps `run_config` (its warnings stay empty, and `unread_span_ids` names the spans not read), and `save_artifact("novelty", ...)` refuses a record with `status` or `partial`. `skill shorten` still writes nothing at the cap.

dokimi/docs/skill-novelty.md

@@ -846,8 +846,8 @@ between repeats keeps its picks but is left out of `noise`. The lexical arm is c
 Treat a `partial` file as a prefix of the run, not a measurement of the whole field: the skills
 and queries reached first are over-represented. To finish, raise `--max-usd` and rerun with
 `DOKIMI_RESPONSE_CACHE` set (it must also have been set on the capped run), which replays the
-calls already paid for at no cost. `skill novelty analyze` and `skill shorten` still write nothing
-when the cap stops them; `atoms` and `probe` keep the calls they made as a partial result.
+calls already paid for at no cost. `skill shorten` still writes nothing when the cap stops it;
+`atoms`, `probe` and `analyze` keep the calls they made as a partial result.
 
 Three properties worth knowing before quoting either:
 
@@ -1110,7 +1110,7 @@ from the network before it falls back to the copy bundled with the package; on a
 with no network that is a pause of a few seconds and a warning. Setting
 `LITELLM_LOCAL_MODEL_COST_MAP=True` skips the fetch and uses the bundled prices.
 
-**Spend cap on `atoms` and `probe`.** `--max-usd` stops these at the limit and keeps the work it
+**Spend cap on `atoms`, `probe` and `analyze`.** `--max-usd` stops these at the limit and keeps the work it
 paid for. `atoms` writes `<stem>.partial.json` beside the `--out` file (`atoms.json` by default;
 the spans read so far, the rest marked unread), and `probe` does the same beside its `--out`
 file (`probes.json` by default; the answers so far). Both exit with a message naming the file. A partial file is never read as a
@@ -1120,7 +1120,29 @@ the summaries computed over the answers the cap left (`paired`, `arms`, `abstent
 confidence blocks, and `judge2`: the second judge shares the cap, so it cannot regrade at all once
 the cap stops the run).
 A partial file is for inspection; rerun to get a finished one.
-`analyze` and `shorten` still lose their work at the cap (a follow-up). Before its first paid
+`analyze` does the same beside its `novelty.json` (`<out stem>.partial.json`, never over a
+finished file; the `--out` file, `novelty.json` in `--out-dir` by default). Its partial file has
+`"status": "partial"` and a `partial` block: `reason` (`spend_cap`, or `no_price` for a call with
+no known price), `stage` (`extraction`, `verify`, `need_graph`, `probes`, `measurement`, `classify`
+or `attribution`: the stage that was running), `message`, `cause` (the cap's own text, with what was
+spent), and `spans_read` / `spans_total` for an extraction stop or `calls_done` / `calls_planned`
+(probe answers) once the measurement started. It keeps the `extracted_atoms` and `source_spans`
+(the unread spans are in `extraction.unread_span_ids`), the `probes` once built (with the
+wordings and answer keys in `probes.items`, so a stop before the first answer still keeps them) and
+the `runs` (probe answers and judgments) so far; it has no `paired`, `arms`, `k0` or view summaries, and no
+`novelty.md` or annotated view is written. The exit is non-zero (1), the message names the file and
+how to get the paid calls back: with `DOKIMI_RESPONSE_CACHE` set to a writable directory on the capped run, a rerun with a
+higher `--max-usd` and the same cache replays them at $0 (a reply that was cut off or unusable is
+asked again; after a `no_price` stop the message says to price the model or drop `--max-usd`
+instead, since a higher cap does not help); without it, or when it could not be written, the message says the calls were not cached. `novelty view`, `subjects`
+(`--novelty-artifact`) and every other reader of a `novelty.json` refuse a partial file by name,
+and `runs list` does not list it (it opens with `status`, not `schema_version`). A record
+with `status: "partial"` but no `partial` block is refused too, and so is saving one with
+`save_artifact("novelty", ...)`. Its `atoms.n` counts the probed sample once the cap's sample is
+chosen, as a finished record does, and every extracted atom for a stop in extraction (`atoms.sampling`
+is then `none`, and `run_config` holds the settings of the run but not its tier). A library caller of
+`dokimi.novelty.analyze()` checks `result.partial` before it saves `result.artifact`.
+`shorten` still loses its work at the cap (a follow-up). Before its first paid
 call `analyze` prints the run's expected cost at list prices to stderr.
 
 The estimate payload has `expected_calls`, the point figure between

dokimi/dokimi/artifacts.py

@@ -212,9 +212,19 @@ def schema_errors(schema: str | Path, instance: Any) -> list[str]:
     ]
 
 
+def summary_errors(kind: str, data: dict) -> list[str]:
+    """What the schema file of ``kind`` leaves unchecked: a finished ``novelty`` record (one without
+    ``status``) needs its ``paired``, ``k0`` and ``spans`` summaries, which only a partial record lacks."""
+    if kind != "novelty" or "status" in data:
+        return []
+    return [f"<root>: {name!r} is a required property"
+            for name in ("paired", "k0", "spans") if name not in data]
+
+
 def validate_artifact(kind: str, data: dict) -> None:
     """Validate ``data`` against the schema for ``kind`` (raises on failure)."""
     errors = schema_errors(_schema_path(kind, schema_version_for(kind, data)), data)
+    errors += summary_errors(kind, data)
     if errors:
         details = "\n".join(f"  - {e}" for e in errors)
         raise ArtifactValidationError(f"{kind} artifact validation failed:\n{details}")
@@ -238,6 +248,9 @@ def save_artifact(kind: str, data: dict, path: Path | str, *, ensure_ascii: bool
       input kind. Values already in ``data`` under these keys are replaced: the file records the
       process that last saved it, so an artifact built from a loaded one takes the new run's id.
       ``data`` itself is not mutated.
+    - A ``novelty`` payload with ``status`` or ``partial`` (what a ``--max-usd`` stop keeps) is
+      refused with :class:`~dokimi.utils.InputError`, as :func:`load_artifact` refuses it: nothing
+      is written, since ``runs`` would list the file as a finished run.
     - The payload is validated first. :class:`ArtifactValidationError` means nothing was
       written at ``path`` (a file already there is untouched); the rejected payload is kept
       at ``<name>.rejected`` so a paid run is not lost to a schema problem. A later valid
@@ -261,6 +274,10 @@ def save_artifact(kind: str, data: dict, path: Path | str, *, ensure_ascii: bool
     """
     if kind not in SCHEMA_VERSIONS:
         raise KeyError(f"Unknown artifact kind: {kind!r}")
+    if _is_partial_novelty(kind, data):  # `runs` would list it as a finished run
+        raise InputError(f"not written to {path}: a partial novelty record (`status` or `partial` "
+                         "set) is not a finished analysis; `skill novelty analyze` keeps it in "
+                         "<out stem>.partial.json")
     path = Path(path)
     path.parent.mkdir(parents=True, exist_ok=True)
     try:
@@ -306,6 +323,12 @@ def _write_target(path: Path) -> Path:
     return target
 
 
+def _is_partial_novelty(kind: str, data: dict) -> bool:
+    """Whether ``data`` is what `skill novelty analyze` keeps when --max-usd stops it: not a finished
+    analysis. Either marker alone is enough: the schema lets a record carry one without the other."""
+    return kind == "novelty" and ("partial" in data or data.get("status") == "partial")
+
+
 def load_artifact(kind: str, path: Path | str, *, validate: bool = True) -> dict:
     """Read and (optionally) validate an artifact of ``kind`` from ``path``.
 
@@ -315,6 +338,13 @@ def load_artifact(kind: str, path: Path | str, *, validate: bool = True) -> dict
     data = read_json_input(path)
     if not isinstance(data, dict):
         raise InputError(f"{path} is not a {kind} artifact: expected a JSON object, got {type(data).__name__}")
+    if _is_partial_novelty(kind, data):
+        block = data.get("partial")
+        reason = block.get("reason") if isinstance(block, dict) else None
+        raise InputError(f"{path} is a partial novelty artifact: the run was stopped before it "
+                         "finished, so it is not a finished analysis. " + (
+                             "Drop --max-usd to run unmetered" if reason == "no_price" else
+                             "Rerun with a higher --max-usd, or without the cap"))
     if validate:
         try:
             validate_artifact(kind, data)

dokimi/dokimi/caching.py

@@ -455,4 +455,16 @@ def store_response(key: str, text: str, usage: UsageRecord) -> None:
         # half-written entry.
         tmp.replace(directory / f"{key}.json")
     except OSError as exc:
+        _failed_stores.add(directory)
         logger.warning("Could not write response cache entry: %s", exc)
+
+
+#: Cache directories a write failed in during this process (a full disk, a quota): a directory
+#: that passes ``os.access`` can still have kept nothing.
+_failed_stores: set[Path] = set()
+
+
+def response_store_failed(directory: Path) -> bool:
+    """Whether a :func:`store_response` write into ``directory`` failed in this process, so some
+    call it paid for may not be in the cache."""
+    return directory in _failed_stores

dokimi/dokimi/commands/novelty.py

@@ -19,7 +19,7 @@
 
 from dokimi.commands.inputs import placebo_text_for_arms_or_exit
 from dokimi.commands.novelty_run import (add_judge2_args, add_spend_cap, judge2_client,
-                                         stamp, stops_at_cap)
+                                         exit_if_analyze_partial, stamp, stops_at_cap)
 from dokimi.commands.novelty_routing_inputs import (
     checked_out_path,
     load_all_vs_all_field,
@@ -139,10 +139,10 @@ def cmd_novelty_analyze(args: argparse.Namespace) -> None:
         raise InputError(str(exc)) from exc
     except (ExtractionError, ValueError) as exc:  # a model failure under way
         raise SystemExit(str(exc)) from None
-
-    stamp(result.artifact, args, result.measure_result, answerer=answerer, judge=judge, judge2=judge2)
     out_dir = Path(args.out_dir) if args.out_dir else _default_dir(args.artifact)
     out_json = Path(args.out_json) if args.out_json else out_dir / "novelty.json"
+    exit_if_analyze_partial(result, out_json)  # a run the cap stopped keeps what it paid for
+    stamp(result.artifact, args, result.measure_result, answerer=answerer, judge=judge, judge2=judge2)
     save_artifact("novelty", result.artifact, out_json)
     out_md = out_json.with_suffix(".md")
     out_md.write_text(render_novelty_markdown(result.artifact), encoding="utf-8")

dokimi/dokimi/commands/novelty_budget.py

@@ -3,15 +3,16 @@
 
 """What a run that ``--max-usd`` stops leaves behind, and what a run is expected to cost.
 
-A stopped atoms or probe run is kept as ``<out stem>.partial.json``, beside the file it would
-have written: a partial result never replaces a finished one, and the readers that take an
-atoms file refuse a partial one.
+A stopped atoms, probe or analyze run is kept as ``<out stem>.partial.json``, beside the file it
+would have written: a partial result never replaces a finished one, and the readers that take an
+atoms file, a probe file or a ``novelty.json`` refuse a partial one.
 """
 
 from __future__ import annotations
 
 import argparse
 import json
+import os
 import sys
 from collections.abc import Callable
 from pathlib import Path
@@ -42,9 +43,48 @@ def partial_write(out: Path, write: Callable[[Path], object]) -> Path:
     raise RuntimeError("unreachable: the sidecar names never run out")
 
 
-def exit_partial(summary: str, path: Path) -> NoReturn:
+def exit_partial(summary: str, path: Path, note: str = "") -> NoReturn:
     """Stop with a message that names the partial file, as a run the cap stopped must."""
-    raise SystemExit(f"{summary}; the partial result is in {path}")
+    raise SystemExit(f"{summary}; the partial result is in {path}{note}")
+
+
+def replay_hint(reason: str = "spend_cap") -> str:
+    """How a rerun gets the calls this run paid for back at $0: the response cache, which must
+    have been set on the run that paid (every novelty call goes through it). A higher
+    ``--max-usd`` is advised only for a ``spend_cap`` stop: a call with no price is refused
+    whatever the cap, so that stop is cured by a price or by dropping the cap."""
+    from dokimi.caching import RESPONSE_CACHE_ENV, response_cache_dir, response_store_failed
+
+    try:
+        cache = response_cache_dir()
+    except ValueError:  # the variable names a file: no call was cached
+        cache = None
+    # Every paid call has gone through the cache by now: a dir missing or read-only kept none,
+    # and one whose write failed (a full disk) may have lost some.
+    active = (cache is not None and os.access(cache, os.W_OK)
+              and not response_store_failed(cache))
+    if active:
+        rerun = ("a higher --max-usd" if reason == "spend_cap"
+                 else "the model priced, or without --max-usd,")
+        return (f". Rerun with {rerun} and the same {RESPONSE_CACHE_ENV}: the calls "
+                "already paid for replay at $0 (a cut-off or unusable reply is asked again)")
+    return (f". {RESPONSE_CACHE_ENV} was not set on this run or could not be written, so a "
+            "rerun pays for these calls again; set it to a writable directory before the next "
+            "run to replay them at $0")
+
+
+def save_analyze_partial(result, out: Path) -> NoReturn:
+    """Keep what a stopped ``analyze`` paid for beside ``out`` and exit, naming the file.
+
+    ``result`` is a :class:`~dokimi.novelty.analyze.AnalyzeResult` with ``partial`` set. The
+    record goes to the sidecar rules of ``atoms`` and ``probe`` (never over a finished
+    ``novelty.json``).
+    """
+    partial = result.partial
+    cause = partial.get("cause")
+    summary = f"{partial['message']} ({cause})" if cause else partial["message"]
+    exit_partial(summary, write_run(result.artifact, out, partial),
+                 replay_hint(partial["reason"]))
 
 
 def write_run(payload: dict, out: Path, partial: dict | None = None) -> Path:

dokimi/dokimi/commands/novelty_run.py

@@ -22,12 +22,20 @@ def run(args: argparse.Namespace) -> None:
         try:
             command(args)
         except SpendCapReached as exc:
-            # A command that keeps its partial result catches the cap itself and never gets here.
+            # A command that keeps its partial result (atoms, probe, analyze) saves it itself.
             raise SystemExit(f"{exc}; nothing was written") from None
 
     return run
 
 
+def exit_if_analyze_partial(result, out) -> None:
+    """Save and exit when the cap stopped ``analyze`` (``result.partial``); a finished run returns."""
+    from dokimi.commands.novelty_budget import save_analyze_partial
+
+    if result.partial:
+        save_analyze_partial(result, out)
+
+
 def usd_amount(text: str) -> float:
     """A positive, finite USD amount (``0``, negatives, ``nan`` and ``inf`` would disable the cap)."""
     value = float(text)