From 667fd3db5ac4ae67877c055df5b2b39625b754c1 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 01:39:19 -0400 Subject: [PATCH 01/58] Play a scripted conversation and assert a scored debrief in the real-model smoke MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The nightly real-model harness previously proved one thing: a single NPC turn came back inside a latency budget. That leaves the two most valuable signals untested — whether a real model can sustain a multi-turn conversation, and whether the pipeline turns one into a *scored* debrief — and when it did fail, the output did not say whether the model failed to download, the server crashed, or the product rejected the model's output. The harness now runs the full journey and classifies every failure: - Scripted multi-turn conversation (the same three player turns as the fake-runtime playthrough in tests/e2e, so both cover the same shape), followed by session end and debrief generation. - Asserts a scored debrief: at least one rubric dimension scored, a numeric overall_score in [0, 100], a non-empty summary. Per-turn parse flags from the debug endpoint catch a conversation that only *looks* real — if every generated turn fell back to the canned safe utterance, the run fails. - Every failure gets one of six classes, each with its own exit code, banner, remedy and report field: budget (1), download (2), checksum (3), runtime (4), pipeline (5), timeout (6). A crashed child process outranks the client-side symptom it caused, so a dead core reads as `runtime`, not `pipeline`. - Checksum drift fails loudly: the on-disk file is re-verified before the weights are loaded, so a drifted or truncated *cache hit* is caught instead of being fed to the smoke. A mismatch deletes the bad local file. - A wall-clock deadline (default 25 min) trips before the CI job timeout so the failure names the phase that ran long, and caps each request timeout to the budget that is left. - Registry resolution moves into the script (--print-registry-model), so the cache key, download URL and expected checksum all come from one lookup. - The run writes a GitHub step summary (verdict, class, remedy, measured latencies, phase durations) alongside the JSON report artifact. The headline full_response_ms is now the median of the scripted turns, so one unlucky turn cannot flap the nightly. session_start_ms and debrief_ms are measured but not budget-checked: the NPC opening is authored scenario text rather than a generation (the old script's comments claimed otherwise), and debrief latency has no documented SLO. tests/scripts/test_nightly_model_smoke.py covers the harness's decision logic with no model, network or llama-server: checksum verification, the end-to-end assertions, budget arithmetic, failure classification, reporting, and the full run orchestration against faked-out servers. Writing it caught a real bug — Deadline.finish() checked the budget, so closing the books on an already exhausted deadline raised out of run_smoke's finally block and threw the classified verdict away, turning a reportable timeout into a bare traceback. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 1170 +++++++++++++++++---- tests/scripts/__init__.py | 1 + tests/scripts/test_nightly_model_smoke.py | 816 ++++++++++++++ 3 files changed, 1771 insertions(+), 216 deletions(-) mode change 100644 => 100755 scripts/nightly-model-smoke.py create mode 100644 tests/scripts/__init__.py create mode 100644 tests/scripts/test_nightly_model_smoke.py diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py old mode 100644 new mode 100755 index 11e94b19..42dbd898 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1,42 +1,83 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: Apache-2.0 -"""Nightly real-model smoke test for latency budget regression. - -Downloads the pinned starter model (if not already cached), starts a local -llama-server (llama-cpp-python's OpenAI-compatible server) against it, starts -convsim-core wired to that server, runs a scripted end-to-end session through -convsim-core's REST API, measures full-response latency, and fails if it -exceeds the documented budget multiplied by ``--ci-hardware-factor``. - -convsim-core's turn endpoint is synchronous — it returns the completed NPC +"""Nightly real-model smoke test (issue #457). + +Exercises the whole play pipeline against a **real local model** — no fake +runtime anywhere — and fails with a *classified* verdict so a red nightly is +triageable from the job summary alone. + +What a full run does +-------------------- +1. **download** — fetch the smallest registry model (role ``starter``: + Qwen3 4B Instruct Q4_K_M) unless the CI cache already holds it. +2. **checksum** — hash the on-disk file and compare it to the registry + ``sha256``. This runs on cache hits too, so a drifted or truncated cached + file fails loudly instead of silently feeding a corrupt model to the smoke. +3. **runtime** — start llama-cpp-python's OpenAI-compatible server on the GGUF + and start ``convsim-core`` wired to it (``CONVSIM_RUNTIME_ID=llama_cpp``). +4. **conversation** — create and start a session on the built-in + ``behavioral_interview`` scenario (the NPC opening is authored scenario text, + not a generation), then submit a scripted multi-turn player conversation. + Every NPC *reply* is generated by the real model under the turn-output JSON + schema. +5. **debrief** — end the session and generate the debrief, asserting a *scored* + debrief comes back (non-empty rubric dimensions, numeric overall score, + non-empty summary). +6. **budget** — compare measured latencies against the documented budgets + scaled by ``--ci-hardware-factor``. + +``convsim-core``'s turn endpoint is synchronous — it returns the completed NPC turn in a single response rather than streaming ``npc.token`` events (token streaming is added by the ``apps/api`` gateway, which is not exercised here). The smoke therefore measures full end-to-end turn latency (which subsumes -time-to-first-token) against the ``FULL_RESPONSE_MS`` budget. - -A budget regression is defined as measured_ms > ci_budget_ms * 1.20. +time-to-first-token) against the ``full_response_ms`` budget. + +Failure classification +---------------------- +Every failure is attributed to one class, printed in a banner, written to the +JSON report (``failure_class``) and the GitHub step summary, and mapped to a +distinct exit code: + +=========== ==== ======================================================= +class exit meaning +=========== ==== ======================================================= +``budget`` 1 latency regression: the pipeline worked, just too slowly +``download`` 2 model could not be fetched (network / HTTP / missing cache) +``checksum`` 3 SHA-256 drift: on-disk bytes ≠ registry ``sha256`` +``runtime`` 4 llama-server or convsim-core crashed, hung, or 5xx'd +``pipeline`` 5 servers healthy but an end-to-end assertion failed +``timeout`` 6 wall-clock budget exhausted (phase named in the message) +=========== ==== ======================================================= + +Transcript content is fully scripted (see ``SCRIPTED_PLAYER_TURNS``) and holds +no user data, so short NPC excerpts are logged and reported — they are the +evidence that a real model, not a canned fallback, drove the conversation. +``save_transcript`` is false, so no transcript file is written. Usage ----- -Download only (called from CI before the model cache is populated):: +Resolve the starter model from the registry (writes ``$GITHUB_OUTPUT``):: - python scripts/nightly-model-smoke.py \\ - --download-only \\ - --model-url \\ - --model-sha256 \\ - --model-id + python scripts/nightly-model-smoke.py --print-registry-model starter + +Download only (cache miss):: + + python scripts/nightly-model-smoke.py --download-only \\ + --model-id --model-url --model-sha256 + +Verify an already-cached file without running the smoke:: + + python scripts/nightly-model-smoke.py --verify-only \\ + --model-id --model-sha256 -Full smoke (model already on disk):: +Full smoke:: python scripts/nightly-model-smoke.py \\ --model-id qwen3-4b-instruct-q4_k_m \\ - --ci-hardware-factor 6 \\ + --model-sha256 \\ + --ci-hardware-factor 20 \\ + --wall-clock-budget-s 1500 \\ --report-path /tmp/smoke-report.json - -Exit codes ----------- -0 All latency budgets met (within 20 % CI tolerance). -1 One or more budgets exceeded, or the server failed to start / respond. """ from __future__ import annotations @@ -45,6 +86,7 @@ import hashlib import json import os +import statistics import subprocess import sys import tempfile @@ -54,20 +96,27 @@ import urllib.request from collections import deque from pathlib import Path -from typing import Optional +from typing import Any, Dict, List, Optional, Sequence REPO_ROOT = Path(__file__).parent.parent - -# Documented latency budgets (ms) from packages/shared/src/types/metrics.ts. -# Keep in sync manually; the script is intentionally not importing TS source. -# Only full_response_ms is measurable through convsim-core's synchronous REST -# turn API; TTFT requires token streaming (apps/api gateway), which this smoke -# does not stand up. +REGISTRY_PATH = REPO_ROOT / "model-registry" / "registry.yaml" + +# Documented latency budgets (ms) from docs/performance.md and the +# LATENCY_BUDGETS constant in packages/shared/src/types/metrics.ts. +# Keep in sync manually; the script intentionally does not import TS source. +# +# Only full_response_ms is budget-checked here: +# - TTFT needs token streaming (the apps/api gateway), which this smoke does +# not stand up, and full_response_ms subsumes it; +# - session_start_ms and debrief_ms are measured and reported but NOT checked. +# The NPC opening is authored scenario text, not a generation, so session +# start is not inference-bound and the CI hardware factor does not apply to +# it; debrief generation has no documented budget (docs/performance.md). BUDGETS_MS = { "full_response_ms": 10_000, } -# Regression tolerance: measured may exceed CI budget by at most this factor. +# Regression tolerance: measured may exceed the CI budget by at most this factor. REGRESSION_TOLERANCE = 1.20 MODELS_DIR = Path.home() / ".convsim" / "models" / "llm" @@ -76,74 +125,320 @@ LLAMA_SERVER_PORT = 7356 # convsim-core's default CONVSIM_LLAMA_CPP_BASE_URL port CORE_PORT = 7399 -# A simple scripted player turn that any instruct model can answer in <30 tokens. -SMOKE_PLAYER_TURN = "Reply with exactly one sentence: Hello, I am ready." +# Wall-clock budget for the whole run, in seconds. Deliberately below the +# workflow's timeout-minutes so the script — which knows which phase it is in — +# fails first with an attributed verdict, instead of GitHub killing the job +# with an unattributable "The operation was canceled". See docs/real-model-smoke.md. +DEFAULT_WALL_CLOCK_BUDGET_S = 1500.0 # 25 min + +# Scripted player turns — deliberately the same script as the fake-runtime +# playthrough in tests/e2e/test_scripted_playthrough.py, so the real-model +# nightly and the per-PR smoke exercise the same conversation shape. +SCRIPTED_PLAYER_TURNS = ( + "I have five years of experience in software development.", + "My biggest achievement was building a cross-team API platform.", + "I believe in clear documentation and knowledge sharing.", +) + +SCENARIO_ID = "behavioral_interview" + +# Longest NPC excerpt copied into the log / report (scripted content only). +EXCERPT_CHARS = 160 + +# A debrief summary shorter than this is not a usable coaching summary. +MIN_SUMMARY_CHARS = 20 + + +# ── Failure classification ──────────────────────────────────────────────────── + + +class FailureClass: + """Mutually exclusive failure classes, each with its own exit code.""" + + BUDGET = "budget" + DOWNLOAD = "download" + CHECKSUM = "checksum" + RUNTIME = "runtime" + PIPELINE = "pipeline" + TIMEOUT = "timeout" + + +EXIT_CODES = { + FailureClass.BUDGET: 1, + FailureClass.DOWNLOAD: 2, + FailureClass.CHECKSUM: 3, + FailureClass.RUNTIME: 4, + FailureClass.PIPELINE: 5, + FailureClass.TIMEOUT: 6, +} + +# What a human should do first for each class. Printed in the failure banner +# and the step summary so triage does not require reading this file. +REMEDIES = { + FailureClass.BUDGET: ( + "Latency regression, not a broken pipeline. Compare measured_ms in the " + "report artifact against recent nightlies. If the runner class or the " + "starter model changed, re-calibrate --ci-hardware-factor; otherwise " + "look for a prompt-size or sampling-parameter regression." + ), + FailureClass.DOWNLOAD: ( + "The model could not be fetched. Check that the registry URL is still " + "live (scripts/validate-registry.py --url-check) and that Hugging Face " + "is reachable. Nothing about the app changed — re-run the job." + ), + FailureClass.CHECKSUM: ( + "SHA-256 drift: the bytes on disk do not match model-registry/registry.yaml. " + "Either the pinned upstream file was replaced (re-pin with " + "scripts/pin-model.py and review the new file) or the cached download is " + "corrupt. The local copy has been deleted, but the Actions cache entry has " + "not: delete it too (`gh cache delete `, or bump the cache-key prefix " + "in model-smoke-nightly.yml) or every re-run restores the same bad bytes. " + "Do NOT relax this check." + ), + FailureClass.RUNTIME: ( + "llama-server or convsim-core crashed, hung, or returned 5xx. The child " + "stderr tail is printed above the banner. Likely causes: runner OOM on " + "model load, an incompatible llama-cpp-python wheel, or an unhandled " + "exception in the turn pipeline." + ), + FailureClass.PIPELINE: ( + "Both servers were healthy but an end-to-end assertion failed: the model " + "ran and produced output the product rejects. Inspect per-turn " + "used_fallback / native-structured-output flags in the report artifact." + ), + FailureClass.TIMEOUT: ( + "The run exhausted its wall-clock budget in the phase named above. " + "Either inference got much slower or a phase hung. Check phase_durations_s " + "in the report artifact before raising --wall-clock-budget-s." + ), +} + + +class SmokeFailure(Exception): + """A failure with a known class, so CI can distinguish *why* the smoke failed.""" + + def __init__(self, failure_class: str, message: str, *, phase: Optional[str] = None) -> None: + super().__init__(message) + self.failure_class = failure_class + self.phase = phase + + @property + def exit_code(self) -> int: + return EXIT_CODES[self.failure_class] + + @property + def remedy(self) -> str: + return REMEDIES[self.failure_class] + + +# ── Wall-clock budget ───────────────────────────────────────────────────────── + + +class Deadline: + """Tracks the run's wall-clock budget and which phase is consuming it.""" + + def __init__(self, budget_s: float) -> None: + self.budget_s = budget_s + self._start = time.monotonic() + self._phase = "startup" + self._phase_start = self._start + self.phase_durations_s: Dict[str, float] = {} + @property + def elapsed_s(self) -> float: + return time.monotonic() - self._start -# ── Download helpers ────────────────────────────────────────────────────────── + @property + def remaining_s(self) -> float: + return self.budget_s - self.elapsed_s + + @property + def phase(self) -> str: + return self._phase + + def enter(self, phase: str) -> None: + """Close out the current phase's duration, start a new one, check the budget.""" + self._close_phase(phase) + self.check() + + def finish(self) -> Dict[str, float]: + """Close the books without checking the budget. + + Called from ``run_smoke``'s ``finally``, where an exhausted budget has + already been classified: raising here would escape the handler and throw + the verdict away, turning a reportable timeout into a bare traceback. + """ + self._close_phase("done") + return self.phase_durations_s + + def _close_phase(self, next_phase: str) -> None: + now = time.monotonic() + self.phase_durations_s[self._phase] = round( + self.phase_durations_s.get(self._phase, 0.0) + (now - self._phase_start), 1 + ) + self._phase = next_phase + self._phase_start = now + + def check(self) -> None: + if self.remaining_s <= 0: + raise SmokeFailure( + FailureClass.TIMEOUT, + f"Wall-clock budget of {self.budget_s:.0f} s exhausted during phase " + f"{self._phase!r} (elapsed {self.elapsed_s:.0f} s)", + phase=self._phase, + ) + + def cap(self, timeout_s: float) -> float: + """Clamp a request timeout to what is left of the budget.""" + self.check() + return max(5.0, min(timeout_s, self.remaining_s)) + + +# ── Registry ────────────────────────────────────────────────────────────────── + + +def resolve_registry_model(role: str, registry_path: Path = REGISTRY_PATH) -> Dict[str, Any]: + """Return ``{id, url, sha256, size_gb}`` for the registry model with ``role``. + + Single source of truth for the workflow: the cache key, the download and the + checksum verification all come from this one lookup, so they cannot drift + apart. ``yaml`` is imported lazily so the module imports without PyYAML. + """ + import yaml # lazy: keeps this module importable for unit tests without PyYAML + + registry = yaml.safe_load(registry_path.read_text(encoding="utf-8")) + matches = [m for m in registry.get("models", []) if m.get("role") == role] + if not matches: + raise SmokeFailure( + FailureClass.DOWNLOAD, + f"No model with role {role!r} in {registry_path}", + ) + if len(matches) > 1: + raise SmokeFailure( + FailureClass.DOWNLOAD, + f"Registry has {len(matches)} models with role {role!r}; expected exactly one", + ) + model = matches[0] + download = model.get("download", {}) + sha256 = download.get("sha256") + url = download.get("url") + if not sha256 or not url: + raise SmokeFailure( + FailureClass.DOWNLOAD, + f"Registry model {model.get('id')!r} is missing download.url or download.sha256", + ) + return { + "id": model["id"], + "url": url, + "sha256": sha256, + "size_gb": model.get("size_gb"), + } + + +def _emit_github_output(values: Dict[str, Any]) -> None: + """Write ``key=value`` lines to $GITHUB_OUTPUT (and always to stdout).""" + lines = [f"{k}={v}" for k, v in values.items()] + for line in lines: + print(line) + out = os.environ.get("GITHUB_OUTPUT") + if out: + with open(out, "a", encoding="utf-8") as f: + f.write("\n".join(lines) + "\n") + + +# ── Download and checksum ───────────────────────────────────────────────────── def _download_with_progress(url: str, dest: Path) -> None: dest.parent.mkdir(parents=True, exist_ok=True) print(f" Downloading {url[:80]}…") req = urllib.request.Request(url, headers={"User-Agent": "convsim-smoke/1.0"}) - with urllib.request.urlopen(req, timeout=600) as resp, open(dest, "wb") as f: - total = int(resp.headers.get("Content-Length", 0)) or None - downloaded = 0 - chunk = 1 << 20 # 1 MB - while True: - buf = resp.read(chunk) - if not buf: - break - f.write(buf) - downloaded += len(buf) - if total: - pct = int(downloaded / total * 100) - print(f"\r {pct}% ({downloaded // 1_048_576} / {total // 1_048_576} MB)", end="", flush=True) - print() - - -def _verify_sha256(path: Path, expected: str) -> None: + try: + with urllib.request.urlopen(req, timeout=600) as resp, open(dest, "wb") as f: + total = int(resp.headers.get("Content-Length", 0)) or None + downloaded = 0 + chunk = 1 << 20 # 1 MB + while True: + buf = resp.read(chunk) + if not buf: + break + f.write(buf) + downloaded += len(buf) + if total: + pct = int(downloaded / total * 100) + print( + f"\r {pct}% ({downloaded // 1_048_576} / {total // 1_048_576} MB)", + end="", + flush=True, + ) + print() + except OSError as exc: + # Network failure, HTTP error, or a full disk — all "could not fetch the + # model", none of which say anything about the app under test. Remove the + # partial file so a re-run does not checksum-fail on a truncated download. + dest.unlink(missing_ok=True) + raise SmokeFailure( + FailureClass.DOWNLOAD, + f"Download failed for {url[:120]}: {exc}", + ) from exc + + +def sha256_file(path: Path) -> str: h = hashlib.sha256() with open(path, "rb") as f: for chunk in iter(lambda: f.read(1 << 20), b""): h.update(chunk) - actual = h.hexdigest() + return h.hexdigest() + + +def verify_model_checksum(path: Path, expected: str, *, delete_on_mismatch: bool = True) -> str: + """Hash ``path`` and compare to ``expected``; raise on drift. + + A mismatch deletes the file by default: leaving a known-bad GGUF in the CI + cache would make every subsequent run fail on the same corrupt bytes, since + the cache key is the expected hash and a cache hit skips the download. + """ + if not path.exists(): + raise SmokeFailure( + FailureClass.DOWNLOAD, + f"Model file not found: {path}. The cache restore produced no model and no " + "download ran — check the cache key and the download step.", + ) + actual = sha256_file(path) if actual != expected: - raise ValueError(f"SHA-256 mismatch: expected {expected}, got {actual}") - print(f" SHA-256 verified: {actual[:16]}…") + size_mb = path.stat().st_size // 1_048_576 + if delete_on_mismatch: + path.unlink(missing_ok=True) + raise SmokeFailure( + FailureClass.CHECKSUM, + f"SHA-256 drift on {path.name} ({size_mb} MB): registry expects " + f"{expected}, on-disk bytes hash to {actual}" + + (". The bad file was deleted." if delete_on_mismatch else ""), + ) + print(f" SHA-256 verified: {actual[:16]}… ({path.stat().st_size // 1_048_576} MB)") + return actual -def download_model(url: str, sha256: str, model_id: str) -> Path: - dest = MODELS_DIR / f"{model_id}.gguf" +def download_model(url: str, sha256: str, model_id: str, models_dir: Path = MODELS_DIR) -> Path: + dest = models_dir / f"{model_id}.gguf" if dest.exists(): print(f" Model already on disk: {dest}") - return dest - _download_with_progress(url, dest) - _verify_sha256(dest, sha256) + else: + _download_with_progress(url, dest) + verify_model_checksum(dest, sha256) return dest # ── Server helpers ──────────────────────────────────────────────────────────── -def _find_model_path(model_id: str) -> Path: - path = MODELS_DIR / f"{model_id}.gguf" - if not path.exists(): - raise FileNotFoundError( - f"Model file not found: {path}\n" - "Run with --download-only first, or check that the model cache is populated." - ) - return path - - def _start_llama_server(model_path: Path, port: int) -> subprocess.Popen: """Start llama-cpp-python's OpenAI-compatible server on the given port. convsim-core's ``llama_cpp`` runtime adapter talks to this server over ``/v1/chat/completions`` (see services/convsim-core/.../runtime/llama_cpp.py). """ - proc = subprocess.Popen( + return subprocess.Popen( [sys.executable, "-m", "llama_cpp.server", "--model", str(model_path), "--host", "127.0.0.1", "--port", str(port), @@ -151,7 +446,6 @@ def _start_llama_server(model_path: Path, port: int) -> subprocess.Popen: stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) - return proc def _start_core(data_dir: Path, port: int, llama_port: int, llama_timeout_s: float) -> subprocess.Popen: @@ -175,23 +469,32 @@ def _start_core(data_dir: Path, port: int, llama_port: int, llama_timeout_s: flo "CONVSIM_PORT": str(port), "CONVSIM_HOST": "127.0.0.1", }) - proc = subprocess.Popen( + return subprocess.Popen( [sys.executable, "-m", "uvicorn", "convsim_core.main:app", "--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"], env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) - return proc -def _wait_for_http(url: str, timeout_s: float, label: str) -> None: - """Poll ``url`` until it returns any HTTP response, or raise on timeout.""" +def _wait_for_http(url: str, timeout_s: float, label: str, proc: Optional[subprocess.Popen] = None) -> None: + """Poll ``url`` until it answers, or raise a RUNTIME failure. + + When ``proc`` is given, an exited child short-circuits the wait: a crashed + server is a runtime failure we can name immediately rather than something to + keep polling for until the timeout. + """ deadline = time.monotonic() + timeout_s last_err: Optional[Exception] = None while time.monotonic() < deadline: + if proc is not None and proc.poll() is not None: + raise SmokeFailure( + FailureClass.RUNTIME, + f"{label} exited with code {proc.returncode} before becoming ready", + ) try: - with urllib.request.urlopen(url, timeout=2): + with urllib.request.urlopen(url, timeout=5): return except urllib.error.HTTPError: # A response (even 4xx) means the server is up and routing. @@ -199,30 +502,54 @@ def _wait_for_http(url: str, timeout_s: float, label: str) -> None: except Exception as exc: # noqa: BLE001 — connection refused while booting last_err = exc time.sleep(1) - raise TimeoutError(f"{label} did not become ready within {timeout_s:.0f} s ({last_err})") + raise SmokeFailure( + FailureClass.RUNTIME, + f"{label} did not become ready within {timeout_s:.0f} s ({last_err})", + ) -# ── Session helpers ─────────────────────────────────────────────────────────── +# ── HTTP helpers (every transport-level failure is a RUNTIME failure) ────────── -def _post_json(url: str, payload: dict, timeout: float = 30.0) -> dict: - data = json.dumps(payload).encode() +def _request_json( + url: str, + *, + payload: Optional[dict] = None, + timeout: float, + expect: Sequence[int] = (200, 201), +) -> dict: + data = json.dumps(payload).encode() if payload is not None else None req = urllib.request.Request( url, data=data, - headers={"Content-Type": "application/json"}, - method="POST", + headers={"Content-Type": "application/json"} if data is not None else {}, + method="POST" if data is not None else "GET", ) - with urllib.request.urlopen(req, timeout=timeout) as resp: - return json.loads(resp.read()) - - -def _get_json(url: str, timeout: float = 30.0) -> dict: - with urllib.request.urlopen(url, timeout=timeout) as resp: - return json.loads(resp.read()) - - -def _npc_turn_content(events: list) -> str: + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + body = json.loads(resp.read()) + if resp.status not in expect: + raise SmokeFailure( + FailureClass.PIPELINE, + f"{url} returned HTTP {resp.status}, expected one of {list(expect)}", + ) + return body + except urllib.error.HTTPError as exc: + detail = exc.read(2048).decode(errors="replace") + # 5xx is the server breaking; 4xx is the server rejecting what we asked + # for, i.e. an end-to-end assertion about the API contract. + cls = FailureClass.RUNTIME if exc.code >= 500 else FailureClass.PIPELINE + raise SmokeFailure(cls, f"{url} returned HTTP {exc.code}: {detail[:500]}") from exc + except SmokeFailure: + raise + except Exception as exc: # noqa: BLE001 — connection reset, timeout, bad JSON + raise SmokeFailure( + FailureClass.RUNTIME, + f"{url} did not answer with usable JSON after {timeout:.0f} s: {exc!r}", + ) from exc + + +def _npc_turn_content(events: Sequence[dict]) -> str: """Return the NPC utterance from a turn/start response's events, or ''.""" for ev in events: if ev.get("event_type") in ("npc_turn", "npc_opening"): @@ -230,10 +557,199 @@ def _npc_turn_content(events: list) -> str: return "" +def _excerpt(text: str) -> str: + collapsed = " ".join(text.split()) + return collapsed[:EXCERPT_CHARS] + ("…" if len(collapsed) > EXCERPT_CHARS else "") + + +# ── Assertions (pure, unit-tested) ──────────────────────────────────────────── + + +def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: + """Check the conversation really came from the model. Returns (failures, warnings). + + The fallback policy is applied only to turns marked ``model_generated``: the + NPC opening is authored scenario text, so counting it would make the + "every turn fell back" check unreachable. + + An empty NPC utterance is always a failure. A single fallback turn is a + warning — a 4 B model does occasionally emit output the validator rejects, + and the product's job is to recover from that — but if *every* generated turn + fell back to the canned safe utterance then no real NPC turns were produced + and the smoke proved nothing, so that is a failure. + """ + failures: List[str] = [] + warnings: List[str] = [] + if not turns: + return (["No turns were recorded; the conversation never ran"], warnings) + + for turn in turns: + label = turn.get("label", f"turn {turn.get('turn_number')}") + if not turn.get("npc_excerpt"): + failures.append(f"{label}: completed but returned no NPC utterance") + + generated = [t for t in turns if t.get("model_generated")] + if not generated: + failures.append("No model-generated NPC turns were recorded") + return (failures, warnings) + + fallbacks = [t for t in generated if t.get("used_fallback")] + if len(fallbacks) == len(generated): + failures.append( + f"All {len(generated)} model-generated NPC turns fell back to the canned " + "safe utterance; the real model produced no usable turn output" + ) + elif fallbacks: + warnings.append( + f"{len(fallbacks)} of {len(generated)} model-generated NPC turns fell back " + "to the safe utterance" + ) + return (failures, warnings) + + +def evaluate_debrief(debrief: Optional[Dict[str, Any]]) -> tuple[List[str], List[str]]: + """Check that a *scored* debrief was produced. Returns (failures, warnings).""" + failures: List[str] = [] + warnings: List[str] = [] + if not debrief: + return (["No debrief was produced"], warnings) + + scores = debrief.get("scores") + if not isinstance(scores, dict) or not scores: + failures.append( + "Debrief has no rubric dimension scores; the model emitted no scorable " + "rubric_observations across the whole conversation" + ) + + overall = debrief.get("overall_score") + if not isinstance(overall, (int, float)) or isinstance(overall, bool): + failures.append(f"Debrief overall_score is not numeric (got {overall!r})") + elif not 0 <= overall <= 100: + failures.append(f"Debrief overall_score {overall} is outside [0, 100]") + + summary = debrief.get("summary") or "" + if len(summary.strip()) < MIN_SUMMARY_CHARS: + failures.append( + f"Debrief summary is shorter than {MIN_SUMMARY_CHARS} characters " + f"({len(summary.strip())} chars)" + ) + + if debrief.get("used_fallback"): + # The scores are still real (they come from per-turn rubric observations), + # so this degrades the debrief narrative rather than invalidating the run. + warnings.append("Debrief narrative used the deterministic fallback, not model prose") + if not debrief.get("turning_points"): + warnings.append("Debrief contains no turning points") + return (failures, warnings) + + +def evaluate_budgets( + measured_ms: Dict[str, float], + ci_hardware_factor: float, + budgets_ms: Dict[str, int] = BUDGETS_MS, + tolerance: float = REGRESSION_TOLERANCE, +) -> tuple[List[str], List[str]]: + """Compare measurements to scaled budgets. Returns (failures, report_lines).""" + failures: List[str] = [] + lines: List[str] = [] + for key, budget_ms in budgets_ms.items(): + measured = measured_ms.get(key) + if measured is None: + continue + ci_budget = budget_ms * ci_hardware_factor + ceiling = ci_budget * tolerance + status = "PASS" if measured <= ceiling else "FAIL" + lines.append( + f"{key}: {measured:.0f} ms " + f"(CI budget {ci_budget:.0f} ms × {tolerance} = {ceiling:.0f} ms) → {status}" + ) + if status == "FAIL": + failures.append( + f"{key} = {measured:.0f} ms exceeds CI ceiling {ceiling:.0f} ms " + f"(documented budget {budget_ms} ms × {ci_hardware_factor} × {tolerance})" + ) + return (failures, lines) + + +# ── Reporting ───────────────────────────────────────────────────────────────── + + +def render_step_summary(results: Dict[str, Any]) -> str: + """Render the GitHub step summary: verdict, class, timings, what to do next.""" + verdict = results.get("verdict", "unknown") + icon = {"pass": "✅", "fail": "❌"}.get(verdict, "⚠️") + lines = [ + f"## {icon} Real-model smoke: {verdict.upper()}", + "", + f"- **Model:** `{results.get('model_id')}`", + f"- **Wall clock:** {results.get('wall_clock_s', 0):.0f} s " + f"(budget {results.get('wall_clock_budget_s', 0):.0f} s)", + ] + if verdict == "fail": + cls = results.get("failure_class", "unknown") + lines += [ + f"- **Failure class:** `{cls}` (exit {results.get('exit_code')})", + "", + "### Failures", + *[f"- {f}" for f in results.get("failures", [])], + "", + "### What to do", + REMEDIES.get(cls, "No remedy recorded."), + ] + measured = results.get("measured_ms") or {} + if measured: + lines += ["", "### Measured latency", "", "| Metric | Measured | CI ceiling |", "|---|---|---|"] + factor = results.get("ci_hardware_factor", 1.0) + for key, value in measured.items(): + budget = BUDGETS_MS.get(key) + ceiling = ( + f"{budget * factor * REGRESSION_TOLERANCE:.0f} ms" if budget else "—" + ) + lines.append(f"| `{key}` | {value:.0f} ms | {ceiling} |") + phases = results.get("phase_durations_s") or {} + if phases: + lines += ["", "### Phase durations", "", "| Phase | Seconds |", "|---|---|"] + lines += [f"| `{k}` | {v:.0f} |" for k, v in phases.items()] + if results.get("warnings"): + lines += ["", "### Warnings", *[f"- {w}" for w in results["warnings"]]] + return "\n".join(lines) + "\n" + + +def _write_step_summary(results: Dict[str, Any]) -> None: + path = os.environ.get("GITHUB_STEP_SUMMARY") + if not path: + return + try: + with open(path, "a", encoding="utf-8") as f: + f.write(render_step_summary(results)) + except OSError as exc: # never let reporting break the verdict + print(f"[smoke] Could not write step summary: {exc}", file=sys.stderr) + + +def _print_banner(failure_class: str, failures: Sequence[str]) -> None: + bar = "━" * 72 + print(f"\n{bar}", file=sys.stderr) + print(f"[smoke] FAILED — class: {failure_class.upper()} " + f"(exit {EXIT_CODES[failure_class]})", file=sys.stderr) + print(bar, file=sys.stderr) + for f in failures: + print(f" ✗ {f}", file=sys.stderr) + print(f"\n → {REMEDIES[failure_class]}", file=sys.stderr) + print(f"{bar}\n", file=sys.stderr) + + +def _write_report(report_path: Optional[Path], results: Dict[str, Any]) -> None: + if not report_path: + return + report_path.parent.mkdir(parents=True, exist_ok=True) + report_path.write_text(json.dumps(results, indent=2), encoding="utf-8") + print(f"[smoke] Report written to {report_path}") + + # ── Smoke run ───────────────────────────────────────────────────────────────── -def _drain(stream: object, sink: Optional["deque"] = None) -> None: +def _drain(stream: object, sink: Optional[deque] = None) -> None: """Consume a subprocess pipe in the background to avoid a full-buffer deadlock. When ``sink`` (a bounded deque) is given, the most recent lines are retained @@ -250,49 +766,114 @@ def _drain(stream: object, sink: Optional["deque"] = None) -> None: pass +def _dump_stderr_tails(tails: Sequence[tuple[str, deque]]) -> None: + for label, tail in tails: + if tail: + print(f"\n[smoke] ── {label} stderr (last {len(tail)} lines) ──", file=sys.stderr) + for line in tail: + print(f" {label[:4]}| {line}", file=sys.stderr) + + +def _crashed_child(procs: Sequence[tuple[str, Optional[subprocess.Popen]]]) -> Optional[str]: + """Return a description of the first child that has exited, if any.""" + for label, proc in procs: + if proc is not None and proc.poll() is not None: + return f"{label} exited with code {proc.returncode}" + return None + + +def _debug_flags_by_turn(base: str, session_id: str, timeout: float) -> Dict[int, Dict[str, Any]]: + """Fetch per-turn parse flags from the dev debug endpoint, keyed by game turn. + + Best-effort: the debug drawer is a diagnostic, so a failure here must not + turn a healthy smoke red. + """ + try: + debug = _request_json(f"{base}/sessions/{session_id}/debug", timeout=timeout) + except SmokeFailure as exc: + print(f"[smoke] Could not read debug flags ({exc}); continuing.") + return {} + return {t["turn_number"]: t for t in debug.get("turns", [])} + + def run_smoke( model_id: str, ci_hardware_factor: float, report_path: Optional[Path], -) -> bool: - """Run the scripted smoke session and check budgets. Returns True on pass.""" - model_path = _find_model_path(model_id) + *, + model_sha256: Optional[str] = None, + models_dir: Path = MODELS_DIR, + wall_clock_budget_s: float = DEFAULT_WALL_CLOCK_BUDGET_S, +) -> int: + """Run the full scripted smoke. Returns the process exit code.""" + clock = Deadline(wall_clock_budget_s) # The turn must be allowed to run past the CI ceiling before we judge it, so # both the adapter timeout and our own POST timeout are derived from the # full-response ceiling with headroom rather than a fixed 30 s / 180 s. - full_ceiling_s = (BUDGETS_MS["full_response_ms"] * ci_hardware_factor * REGRESSION_TOLERANCE) / 1000 + full_ceiling_s = ( + BUDGETS_MS["full_response_ms"] * ci_hardware_factor * REGRESSION_TOLERANCE + ) / 1000 llama_timeout_s = max(180.0, full_ceiling_s + 60.0) turn_timeout_s = llama_timeout_s + 30.0 - results: dict = { + + results: Dict[str, Any] = { "model_id": model_id, + "scenario_id": SCENARIO_ID, "ci_hardware_factor": ci_hardware_factor, - "budgets_ms": BUDGETS_MS, + "budgets_ms": dict(BUDGETS_MS), "ci_budgets_ms": {k: v * ci_hardware_factor for k, v in BUDGETS_MS.items()}, "regression_tolerance": REGRESSION_TOLERANCE, + "wall_clock_budget_s": wall_clock_budget_s, "measured_ms": {}, + "turns": [], + "debrief": None, "verdict": "pending", + "failure_class": None, + "exit_code": None, "failures": [], + "warnings": [], } - with tempfile.TemporaryDirectory(prefix="convsim-smoke-") as tmp: - data_dir = Path(tmp) - # Bounded tails of each child's stderr, surfaced only if the smoke fails - # so a server-side error is diagnosable from the CI logs. - llama_stderr_tail: deque = deque(maxlen=200) - core_stderr_tail: deque = deque(maxlen=200) + model_path = models_dir / f"{model_id}.gguf" + llama_proc: Optional[subprocess.Popen] = None + core_proc: Optional[subprocess.Popen] = None + llama_stderr_tail: deque = deque(maxlen=200) + core_stderr_tail: deque = deque(maxlen=200) - print(f"\n[smoke] Starting llama-server on port {LLAMA_SERVER_PORT} with model: {model_path.name}") - llama_proc = _start_llama_server(model_path, LLAMA_SERVER_PORT) - threading.Thread(target=_drain, args=(llama_proc.stderr, llama_stderr_tail), daemon=True).start() - threading.Thread(target=_drain, args=(llama_proc.stdout,), daemon=True).start() + try: + # ── checksum ────────────────────────────────────────────────────────── + clock.enter("checksum") + if model_sha256: + print(f"[smoke] Verifying model checksum before use: {model_path}") + verify_model_checksum(model_path, model_sha256) + elif not model_path.exists(): + raise SmokeFailure( + FailureClass.DOWNLOAD, + f"Model file not found: {model_path}. Run with --download-only first, " + "or check that the model cache is populated.", + ) + else: + results["warnings"].append( + "No --model-sha256 given: checksum drift was not checked before the run" + ) + + with tempfile.TemporaryDirectory(prefix="convsim-smoke-") as tmp: + data_dir = Path(tmp) + + # ── runtime ─────────────────────────────────────────────────────── + clock.enter("runtime_start") + print(f"\n[smoke] Starting llama-server on port {LLAMA_SERVER_PORT} " + f"with model: {model_path.name}") + llama_proc = _start_llama_server(model_path, LLAMA_SERVER_PORT) + threading.Thread(target=_drain, args=(llama_proc.stderr, llama_stderr_tail), daemon=True).start() + threading.Thread(target=_drain, args=(llama_proc.stdout,), daemon=True).start() - core_proc: Optional[subprocess.Popen] = None - try: print("[smoke] Waiting for llama-server (model load)…") _wait_for_http( f"http://127.0.0.1:{LLAMA_SERVER_PORT}/v1/models", - timeout_s=300.0, + timeout_s=clock.cap(300.0), label="llama-server", + proc=llama_proc, ) print("[smoke] llama-server ready.") @@ -301,147 +882,304 @@ def run_smoke( threading.Thread(target=_drain, args=(core_proc.stderr, core_stderr_tail), daemon=True).start() threading.Thread(target=_drain, args=(core_proc.stdout,), daemon=True).start() + base = f"http://127.0.0.1:{CORE_PORT}/api" _wait_for_http( - f"http://127.0.0.1:{CORE_PORT}/api/health", - timeout_s=120.0, + f"{base}/health", + timeout_s=clock.cap(120.0), label="convsim-core", + proc=core_proc, ) - base = f"http://127.0.0.1:{CORE_PORT}/api" # Sanity-check that the real runtime is wired (not the fake default). - health = _get_json(f"{base}/health") + health = _request_json(f"{base}/health", timeout=clock.cap(30.0)) runtime_id = health.get("llm_runtime", {}).get("runtime_id") + results["runtime_id"] = runtime_id print(f"[smoke] convsim-core ready (runtime_id={runtime_id}).") if runtime_id != "llama_cpp": - raise RuntimeError( + raise SmokeFailure( + FailureClass.PIPELINE, f"Expected runtime_id 'llama_cpp' but core reports {runtime_id!r}. " - "Check CONVSIM_RUNTIME_ID wiring." + "Check CONVSIM_RUNTIME_ID wiring — the smoke must not run on a fake runtime.", ) - # Create → start → turn against the built-in behavioral_interview scenario. - session = _post_json(f"{base}/sessions", { - "scenario_id": "behavioral_interview", + # ── conversation ────────────────────────────────────────────────── + clock.enter("conversation") + session = _request_json(f"{base}/sessions", payload={ + "scenario_id": SCENARIO_ID, "difficulty": "standard", "player_role_name": "Smoke Tester", "language": "en", "input_mode": "text-only", "tts_enabled": False, "show_state_meters": False, - "save_transcript": False, + "save_transcript": False, # no transcript file written during smoke runs "seed": 42, - }) + }, timeout=clock.cap(30.0)) session_id = session["session_id"] + results["session_id"] = session_id print(f"[smoke] Session {session_id} created. Starting session…") - # /start generates the NPC opening — a real model call, so it needs - # the same generous timeout as the turn on slow CI hardware. - _post_json(f"{base}/sessions/{session_id}/start", {}, timeout=turn_timeout_s) - print("[smoke] NPC opening received. Submitting player turn…") - - # Submit a player turn and time the synchronous end-to-end response. - t_start = time.monotonic() - turn = _post_json( - f"{base}/sessions/{session_id}/turn", - {"content": SMOKE_PLAYER_TURN}, - timeout=turn_timeout_s, + + # /start replays the scenario's authored opening line — no model + # call — so it is measured for the documented session-start metric but + # needs no inference-sized timeout. + t0 = time.monotonic() + start = _request_json( + f"{base}/sessions/{session_id}/start", + payload={}, + timeout=clock.cap(60.0), ) - full_ms = (time.monotonic() - t_start) * 1000 - npc_text = _npc_turn_content(turn.get("events", [])) - print(f"[smoke] Full response: {full_ms:.0f} ms | NPC said: {npc_text[:80]!r}") - if not npc_text: - raise RuntimeError("Turn completed but no npc_turn content was returned") - - results["measured_ms"] = {"full_response_ms": round(full_ms)} - - # Check budgets - failures = [] - for key, budget_ms in BUDGETS_MS.items(): - measured = results["measured_ms"].get(key) - if measured is None: - continue - ci_budget = budget_ms * ci_hardware_factor - ceiling = ci_budget * REGRESSION_TOLERANCE - status = "PASS" if measured <= ceiling else "FAIL" - print( - f" {key}: {measured:.0f} ms " - f"(CI budget {ci_budget:.0f} ms × {REGRESSION_TOLERANCE} = {ceiling:.0f} ms) " - f"→ {status}" + session_start_ms = (time.monotonic() - t0) * 1000 + opening = _npc_turn_content(start.get("events", [])) + print(f"[smoke] NPC opening (authored) in {session_start_ms:.0f} ms: " + f"{_excerpt(opening)!r}") + results["measured_ms"]["session_start_ms"] = round(session_start_ms) + results["turns"].append({ + "label": "npc_opening", + "turn_number": 0, + "model_generated": False, + "latency_ms": round(session_start_ms), + "npc_excerpt": _excerpt(opening), + }) + + turn_latencies: List[float] = [] + for i, player_text in enumerate(SCRIPTED_PLAYER_TURNS, start=1): + clock.check() + print(f"[smoke] Player turn {i}/{len(SCRIPTED_PLAYER_TURNS)}…") + t0 = time.monotonic() + turn = _request_json( + f"{base}/sessions/{session_id}/turn", + payload={"content": player_text}, + timeout=clock.cap(turn_timeout_s), ) - if status == "FAIL": - failures.append( - f"{key} = {measured:.0f} ms exceeds CI ceiling {ceiling:.0f} ms " - f"(budget {budget_ms} ms × {ci_hardware_factor} × {REGRESSION_TOLERANCE})" + latency_ms = (time.monotonic() - t0) * 1000 + turn_latencies.append(latency_ms) + npc_text = _npc_turn_content(turn.get("events", [])) + print(f" {latency_ms:.0f} ms | NPC: {_excerpt(npc_text)!r}") + results["turns"].append({ + "label": f"player_turn_{i}", + "turn_number": i, + "model_generated": True, + "latency_ms": round(latency_ms), + "npc_excerpt": _excerpt(npc_text), + "ending_type": turn.get("ending_type"), + }) + if turn.get("ending_type"): + print(f"[smoke] Scenario ended early (ending_type={turn['ending_type']}); " + "stopping the scripted turns.") + break + + # The median is the headline full-response figure: a single unlucky + # turn (runner steal, a long NPC answer) must not flap the nightly, + # while a real regression moves every turn and so moves the median. + results["measured_ms"]["full_response_ms"] = round(statistics.median(turn_latencies)) + results["measured_ms"]["full_response_max_ms"] = round(max(turn_latencies)) + + # Per-turn parse flags tell us whether the model's structured output + # was accepted or quietly replaced by the safe fallback. + flags = _debug_flags_by_turn(base, session_id, clock.cap(30.0)) + for turn in results["turns"]: + flag = flags.get(turn["turn_number"]) + if flag: + turn["used_fallback"] = bool(flag.get("used_fallback")) + turn["used_native_structured_output"] = bool( + flag.get("used_native_structured_output") ) - results["failures"] = failures - results["verdict"] = "pass" if not failures else "fail" - - except BaseException as exc: - # The client-side error (e.g. HTTP 500 from a turn) says nothing about - # what went wrong inside convsim-core. Surface the captured stderr - # tails so the real traceback is visible in the CI logs, then re-raise. - print(f"\n[smoke] ERROR during run: {exc!r}", file=sys.stderr) - for label, tail in (("convsim-core", core_stderr_tail), ("llama-server", llama_stderr_tail)): - if tail: - print(f"\n[smoke] ── {label} stderr (last {len(tail)} lines) ──", file=sys.stderr) - for line in tail: - print(f" {label[:4]}| {line}", file=sys.stderr) - raise - finally: - for proc in (core_proc, llama_proc): - if proc is None: - continue - proc.terminate() - try: - proc.wait(timeout=10) - except subprocess.TimeoutExpired: - proc.kill() - - if report_path: - report_path.parent.mkdir(parents=True, exist_ok=True) - report_path.write_text(json.dumps(results, indent=2)) - print(f"\n[smoke] Report written to {report_path}") - - if results["failures"]: - print("\n[smoke] FAILED — latency budget regression detected:") - for f in results["failures"]: - print(f" ✗ {f}") - return False - - print(f"\n[smoke] PASSED — all latency budgets met on model {model_id}.") - return True + # ── debrief ─────────────────────────────────────────────────────── + clock.enter("debrief") + print("[smoke] Ending session…") + _request_json(f"{base}/sessions/{session_id}/end", payload={}, timeout=clock.cap(30.0)) + print("[smoke] Generating debrief (real model call)…") + t0 = time.monotonic() + debrief = _request_json( + f"{base}/sessions/{session_id}/debrief", + payload={}, + timeout=clock.cap(turn_timeout_s * 2), + ) + debrief_ms = (time.monotonic() - t0) * 1000 + results["measured_ms"]["debrief_ms"] = round(debrief_ms) + results["debrief"] = { + "outcome": debrief.get("outcome"), + "total_turns": debrief.get("total_turns"), + "scores": debrief.get("scores"), + "overall_score": debrief.get("overall_score"), + "summary_excerpt": _excerpt(debrief.get("summary") or ""), + "strength_count": len(debrief.get("strengths") or []), + "improvement_count": len(debrief.get("improvements") or []), + "turning_point_count": len(debrief.get("turning_points") or []), + "used_fallback": bool(debrief.get("used_fallback")), + "latency_ms": round(debrief_ms), + } + print(f"[smoke] Debrief in {debrief_ms:.0f} ms | " + f"overall_score={debrief.get('overall_score')} " + f"scores={debrief.get('scores')}") + + # ── assertions ──────────────────────────────────────────────────── + clock.enter("assertions") + turn_failures, turn_warnings = evaluate_turns(results["turns"]) + debrief_failures, debrief_warnings = evaluate_debrief(debrief) + results["warnings"] += turn_warnings + debrief_warnings + pipeline_failures = turn_failures + debrief_failures + if pipeline_failures: + results["failures"] = pipeline_failures + raise SmokeFailure( + FailureClass.PIPELINE, + f"{len(pipeline_failures)} end-to-end assertion(s) failed", + ) + + # ── budget ──────────────────────────────────────────────────────── + clock.enter("budget") + budget_failures, budget_lines = evaluate_budgets( + results["measured_ms"], ci_hardware_factor + ) + for line in budget_lines: + print(f" {line}") + if budget_failures: + results["failures"] = budget_failures + raise SmokeFailure( + FailureClass.BUDGET, f"{len(budget_failures)} latency budget(s) exceeded" + ) + + results["verdict"] = "pass" + results["exit_code"] = 0 + + except SmokeFailure as exc: + # A crashed child outranks whatever the client saw: "core exited with + # code 1" is the real story, not the connection reset it caused. + crashed = _crashed_child((("llama-server", llama_proc), ("convsim-core", core_proc))) + failure_class = FailureClass.RUNTIME if crashed else exc.failure_class + results["verdict"] = "fail" + results["failure_class"] = failure_class + results["exit_code"] = EXIT_CODES[failure_class] + results["failed_phase"] = exc.phase or clock.phase + if not results["failures"]: + results["failures"] = [str(exc)] + if crashed: + results["failures"].insert(0, crashed) + _dump_stderr_tails((("convsim-core", core_stderr_tail), ("llama-server", llama_stderr_tail))) + _print_banner(failure_class, results["failures"]) + + # Exception, not BaseException: a Ctrl-C should still stop the children (the + # finally block below) but must not be dressed up as a product failure. + except Exception as exc: + results["verdict"] = "fail" + results["failure_class"] = FailureClass.PIPELINE + results["exit_code"] = EXIT_CODES[FailureClass.PIPELINE] + results["failed_phase"] = clock.phase + results["failures"] = [ + f"Unexpected error in phase {clock.phase!r}: {exc!r} — this is most likely " + "a bug in the smoke harness itself, not in the product " + "(tests/scripts/test_nightly_model_smoke.py covers its logic)" + ] + _dump_stderr_tails((("convsim-core", core_stderr_tail), ("llama-server", llama_stderr_tail))) + _print_banner(FailureClass.PIPELINE, results["failures"]) + + finally: + for proc in (core_proc, llama_proc): + if proc is None: + continue + proc.terminate() + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + proc.kill() + results["phase_durations_s"] = clock.finish() + results["wall_clock_s"] = round(clock.elapsed_s, 1) + + if results["verdict"] == "pass": + generated = sum(1 for t in results["turns"] if t.get("model_generated")) + print(f"\n[smoke] PASSED — real model {model_id} produced " + f"{generated} NPC turn(s) and a scored debrief " + f"(overall_score={(results['debrief'] or {}).get('overall_score')}) " + f"in {results['wall_clock_s']:.0f} s.") + for warning in results["warnings"]: + print(f" ! {warning}") + + _write_report(report_path, results) + _write_step_summary(results) + return int(results["exit_code"] or 0) # ── CLI ─────────────────────────────────────────────────────────────────────── -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter) +def main(argv: Optional[Sequence[str]] = None) -> int: + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--print-registry-model", metavar="ROLE", default=None, + help="Resolve the registry model with this role (e.g. 'starter'), " + "write id/url/sha256 to $GITHUB_OUTPUT, and exit.") parser.add_argument("--download-only", action="store_true", help="Download and verify the model file, then exit.") + parser.add_argument("--verify-only", action="store_true", + help="Verify the on-disk model against --model-sha256, then exit.") parser.add_argument("--model-url", default=None, help="Download URL (required for --download-only).") parser.add_argument("--model-sha256", default=None, - help="Expected SHA-256 hex (required for --download-only).") - parser.add_argument("--model-id", required=True, + help="Expected SHA-256 hex. Required for --download-only/--verify-only; " + "on a full run the model is re-verified before use when given.") + parser.add_argument("--model-id", default=None, help="Registry model ID (e.g. qwen3-4b-instruct-q4_k_m).") + parser.add_argument("--models-dir", default=None, + help=f"Directory holding .gguf (default {MODELS_DIR}).") parser.add_argument("--ci-hardware-factor", type=float, default=1.0, help="Multiplier applied to documented budgets for CI hardware (default 1.0).") - parser.add_argument("--report-path", default=None, - help="Write JSON report to this path.") + parser.add_argument("--wall-clock-budget-s", type=float, default=DEFAULT_WALL_CLOCK_BUDGET_S, + help=f"Fail with exit {EXIT_CODES[FailureClass.TIMEOUT]} if the run " + f"exceeds this many seconds (default {DEFAULT_WALL_CLOCK_BUDGET_S:.0f}).") + parser.add_argument("--report-path", default=None, help="Write JSON report to this path.") - args = parser.parse_args() + args = parser.parse_args(argv) + models_dir = Path(args.models_dir) if args.models_dir else MODELS_DIR - if args.download_only: - if not args.model_url or not args.model_sha256: - parser.error("--model-url and --model-sha256 are required with --download-only") - print(f"[smoke] Downloading model {args.model_id}…") - download_model(args.model_url, args.model_sha256, args.model_id) - return 0 - - report = Path(args.report_path) if args.report_path else None - passed = run_smoke(args.model_id, args.ci_hardware_factor, report) - return 0 if passed else 1 + try: + if args.print_registry_model: + model = resolve_registry_model(args.print_registry_model) + _emit_github_output({ + "model_id": model["id"], + "model_url": model["url"], + "model_sha256": model["sha256"], + "model_size_gb": model["size_gb"], + }) + return 0 + + if not args.model_id: + parser.error("--model-id is required") + + if args.download_only: + if not args.model_url or not args.model_sha256: + parser.error("--model-url and --model-sha256 are required with --download-only") + print(f"[smoke] Downloading model {args.model_id}…") + download_model(args.model_url, args.model_sha256, args.model_id, models_dir) + return 0 + + if args.verify_only: + if not args.model_sha256: + parser.error("--model-sha256 is required with --verify-only") + print(f"[smoke] Verifying cached model {args.model_id}…") + verify_model_checksum(models_dir / f"{args.model_id}.gguf", args.model_sha256) + return 0 + + except SmokeFailure as exc: + _print_banner(exc.failure_class, [str(exc)]) + _write_step_summary({ + "verdict": "fail", + "model_id": args.model_id, + "failure_class": exc.failure_class, + "exit_code": exc.exit_code, + "failures": [str(exc)], + }) + return exc.exit_code + + return run_smoke( + args.model_id, + args.ci_hardware_factor, + Path(args.report_path) if args.report_path else None, + model_sha256=args.model_sha256, + models_dir=models_dir, + wall_clock_budget_s=args.wall_clock_budget_s, + ) if __name__ == "__main__": diff --git a/tests/scripts/__init__.py b/tests/scripts/__init__.py new file mode 100644 index 00000000..98813136 --- /dev/null +++ b/tests/scripts/__init__.py @@ -0,0 +1 @@ +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py new file mode 100644 index 00000000..8b86616b --- /dev/null +++ b/tests/scripts/test_nightly_model_smoke.py @@ -0,0 +1,816 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Unit tests for the nightly real-model smoke harness (issue #457). + +These run on every PR and need no model, no network and no llama-server: they +cover the harness's decision logic — checksum verification, the end-to-end +assertions, budget evaluation, failure classification and reporting — so that +when the nightly goes red, the verdict it prints can be trusted. + +The real-model path itself is exercised only by +.github/workflows/model-smoke-nightly.yml; see docs/real-model-smoke.md. +""" +from __future__ import annotations + +import hashlib +import importlib.util +import json +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] +SCRIPT_PATH = REPO_ROOT / "scripts" / "nightly-model-smoke.py" + + +def _load_module(): + """Import the hyphenated script as a module.""" + spec = importlib.util.spec_from_file_location("nightly_model_smoke", SCRIPT_PATH) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +smoke = _load_module() + + +# --------------------------------------------------------------------------- +# Failure classification contract +# --------------------------------------------------------------------------- + + +class TestFailureClassification: + """Each class maps to its own exit code, so CI can tell failures apart.""" + + def test_every_class_has_a_unique_exit_code(self) -> None: + codes = list(smoke.EXIT_CODES.values()) + assert len(codes) == len(set(codes)), "exit codes must be distinguishable" + assert 0 not in codes, "a failure must never exit 0" + + def test_every_class_has_an_exit_code_and_a_remedy(self) -> None: + classes = { + value for name, value in vars(smoke.FailureClass).items() + if not name.startswith("_") and isinstance(value, str) + } + assert classes == set(smoke.EXIT_CODES) + assert classes == set(smoke.REMEDIES) + + def test_download_and_checksum_are_separate_classes(self) -> None: + # The acceptance criterion for #457: a download failure and a checksum + # drift must not look alike in CI output. + assert smoke.EXIT_CODES[smoke.FailureClass.DOWNLOAD] != smoke.EXIT_CODES[ + smoke.FailureClass.CHECKSUM + ] + + def test_failure_carries_its_exit_code_and_remedy(self) -> None: + exc = smoke.SmokeFailure(smoke.FailureClass.RUNTIME, "boom", phase="runtime_start") + assert exc.exit_code == smoke.EXIT_CODES[smoke.FailureClass.RUNTIME] + assert exc.phase == "runtime_start" + assert exc.remedy + + +# --------------------------------------------------------------------------- +# Checksum verification +# --------------------------------------------------------------------------- + + +class TestChecksumVerification: + """Checksum drift must fail loudly, and must not poison the next run.""" + + @staticmethod + def _write(tmp_path: Path, payload: bytes = b"gguf-ish bytes") -> tuple[Path, str]: + path = tmp_path / "model.gguf" + path.write_bytes(payload) + return path, hashlib.sha256(payload).hexdigest() + + def test_matching_checksum_returns_the_digest(self, tmp_path: Path) -> None: + path, digest = self._write(tmp_path) + assert smoke.verify_model_checksum(path, digest) == digest + assert path.exists() + + def test_drift_raises_checksum_class(self, tmp_path: Path) -> None: + path, _ = self._write(tmp_path) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.verify_model_checksum(path, "0" * 64) + assert exc_info.value.failure_class == smoke.FailureClass.CHECKSUM + assert exc_info.value.exit_code == 3 + + def test_drift_deletes_the_bad_file_so_a_rerun_redownloads(self, tmp_path: Path) -> None: + path, _ = self._write(tmp_path) + with pytest.raises(smoke.SmokeFailure): + smoke.verify_model_checksum(path, "0" * 64) + assert not path.exists() + + def test_drift_can_keep_the_file_for_inspection(self, tmp_path: Path) -> None: + path, _ = self._write(tmp_path) + with pytest.raises(smoke.SmokeFailure): + smoke.verify_model_checksum(path, "0" * 64, delete_on_mismatch=False) + assert path.exists() + + def test_missing_file_is_a_download_failure_not_a_checksum_failure(self, tmp_path: Path) -> None: + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.verify_model_checksum(tmp_path / "absent.gguf", "0" * 64) + assert exc_info.value.failure_class == smoke.FailureClass.DOWNLOAD + + def test_download_skips_the_fetch_but_still_verifies_an_existing_file( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + payload = b"already here" + (tmp_path / "m.gguf").write_bytes(payload) + + def _never(*args: object, **kwargs: object) -> None: + raise AssertionError("must not re-download an existing file") + + monkeypatch.setattr(smoke, "_download_with_progress", _never) + # Correct digest: returns the path. + assert smoke.download_model( + "http://example.invalid/m.gguf", hashlib.sha256(payload).hexdigest(), "m", tmp_path + ) == tmp_path / "m.gguf" + # Wrong digest: a cache hit with drifted bytes still fails. + (tmp_path / "m.gguf").write_bytes(payload) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.download_model("http://example.invalid/m.gguf", "0" * 64, "m", tmp_path) + assert exc_info.value.failure_class == smoke.FailureClass.CHECKSUM + + +# --------------------------------------------------------------------------- +# Registry resolution +# --------------------------------------------------------------------------- + + +class TestRegistryResolution: + """The workflow's cache key, URL and checksum come from one lookup.""" + + def test_resolves_the_real_starter_model(self) -> None: + model = smoke.resolve_registry_model("starter") + assert model["id"] + assert model["url"].startswith("https://") + assert len(model["sha256"]) == 64 + assert model["sha256"] == model["sha256"].lower() + + def test_starter_is_the_smallest_registry_model(self) -> None: + # #457 asks for the smallest registry model; guard against a future + # registry edit that makes "starter" no longer the smallest. + import yaml + + registry = yaml.safe_load( + (REPO_ROOT / "model-registry" / "registry.yaml").read_text(encoding="utf-8") + ) + # The user-supplied placeholder has no size and nothing to download. + sizes = { + m["id"]: m["size_gb"] for m in registry["models"] + if m.get("size_gb") is not None + } + starter = smoke.resolve_registry_model("starter") + assert sizes[starter["id"]] == min(sizes.values()) + + def test_unknown_role_is_a_download_failure(self, tmp_path: Path) -> None: + registry = tmp_path / "registry.yaml" + registry.write_text("models: []\n", encoding="utf-8") + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.resolve_registry_model("starter", registry) + assert exc_info.value.failure_class == smoke.FailureClass.DOWNLOAD + + def test_model_without_checksum_is_rejected(self, tmp_path: Path) -> None: + registry = tmp_path / "registry.yaml" + registry.write_text( + "models:\n" + " - id: x\n" + " role: starter\n" + " download:\n" + " url: https://example.invalid/x.gguf\n", + encoding="utf-8", + ) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.resolve_registry_model("starter", registry) + assert "sha256" in str(exc_info.value) + + def test_github_output_is_appended(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + out = tmp_path / "gh-output" + monkeypatch.setenv("GITHUB_OUTPUT", str(out)) + smoke._emit_github_output({"model_id": "abc", "model_sha256": "def"}) + assert out.read_text(encoding="utf-8").splitlines() == [ + "model_id=abc", + "model_sha256=def", + ] + + +# --------------------------------------------------------------------------- +# Conversation assertions +# --------------------------------------------------------------------------- + + +def _turn(**overrides: object) -> dict: + turn = { + "label": "player_turn_1", + "turn_number": 1, + "model_generated": True, + "npc_excerpt": "Tell me more about that platform.", + "used_fallback": False, + } + turn.update(overrides) + return turn + + +class TestEvaluateTurns: + """Real NPC turns are the whole point of the nightly.""" + + def test_healthy_conversation_passes(self) -> None: + failures, warnings = smoke.evaluate_turns([_turn(), _turn(turn_number=2)]) + assert failures == [] + assert warnings == [] + + def test_no_turns_fails(self) -> None: + failures, _ = smoke.evaluate_turns([]) + assert failures + + def test_empty_npc_utterance_fails(self) -> None: + failures, _ = smoke.evaluate_turns([_turn(npc_excerpt="")]) + assert any("no NPC utterance" in f for f in failures) + + def test_all_generated_turns_falling_back_fails(self) -> None: + turns = [_turn(used_fallback=True), _turn(turn_number=2, used_fallback=True)] + failures, _ = smoke.evaluate_turns(turns) + assert any("fell back" in f for f in failures) + + def test_one_fallback_among_several_only_warns(self) -> None: + turns = [_turn(used_fallback=True), _turn(turn_number=2), _turn(turn_number=3)] + failures, warnings = smoke.evaluate_turns(turns) + assert failures == [] + assert any("fell back" in w for w in warnings) + + def test_authored_opening_does_not_count_as_a_generated_turn(self) -> None: + # The NPC opening is scenario text, not inference. A run whose only + # non-fallback turn is the opening has produced no real NPC turns. + turns = [ + {"label": "npc_opening", "turn_number": 0, "model_generated": False, + "npc_excerpt": "Thanks for coming in."}, + _turn(used_fallback=True), + ] + failures, _ = smoke.evaluate_turns(turns) + assert any("fell back" in f for f in failures) + + def test_opening_only_run_fails(self) -> None: + turns = [{"label": "npc_opening", "turn_number": 0, "model_generated": False, + "npc_excerpt": "Thanks for coming in."}] + failures, _ = smoke.evaluate_turns(turns) + assert any("No model-generated" in f for f in failures) + + +# --------------------------------------------------------------------------- +# Debrief assertions +# --------------------------------------------------------------------------- + + +def _debrief(**overrides: object) -> dict: + doc = { + "scores": {"structure": 56.0, "evidence": 48.0}, + "overall_score": 52.0, + "summary": "You gave concrete examples but hedged on the trade-off question.", + "turning_points": [{"turn_number": 2, "description": "d", "impact": "i"}], + "used_fallback": False, + } + doc.update(overrides) + return doc + + +class TestEvaluateDebrief: + """"Assert a scored debrief is produced" — the #457 acceptance criterion.""" + + def test_scored_debrief_passes(self) -> None: + failures, warnings = smoke.evaluate_debrief(_debrief()) + assert failures == [] + assert warnings == [] + + def test_missing_debrief_fails(self) -> None: + assert smoke.evaluate_debrief(None)[0] + + def test_unscored_debrief_fails(self) -> None: + failures, _ = smoke.evaluate_debrief(_debrief(scores={}, overall_score=None)) + assert any("no rubric dimension scores" in f for f in failures) + assert any("not numeric" in f for f in failures) + + def test_non_numeric_overall_score_fails(self) -> None: + failures, _ = smoke.evaluate_debrief(_debrief(overall_score="52")) + assert any("not numeric" in f for f in failures) + + def test_boolean_overall_score_is_not_a_number(self) -> None: + failures, _ = smoke.evaluate_debrief(_debrief(overall_score=True)) + assert any("not numeric" in f for f in failures) + + @pytest.mark.parametrize("score", [-1, 101]) + def test_out_of_range_overall_score_fails(self, score: float) -> None: + failures, _ = smoke.evaluate_debrief(_debrief(overall_score=score)) + assert any("outside [0, 100]" in f for f in failures) + + def test_empty_summary_fails(self) -> None: + failures, _ = smoke.evaluate_debrief(_debrief(summary=" ")) + assert any("summary is shorter" in f for f in failures) + + def test_fallback_narrative_only_warns_because_scores_are_still_real(self) -> None: + failures, warnings = smoke.evaluate_debrief(_debrief(used_fallback=True)) + assert failures == [] + assert any("fallback" in w for w in warnings) + + def test_absent_turning_points_only_warn(self) -> None: + failures, warnings = smoke.evaluate_debrief(_debrief(turning_points=[])) + assert failures == [] + assert any("turning points" in w for w in warnings) + + +# --------------------------------------------------------------------------- +# Budget evaluation +# --------------------------------------------------------------------------- + + +class TestEvaluateBudgets: + def test_within_budget_passes(self) -> None: + failures, lines = smoke.evaluate_budgets({"full_response_ms": 100_000}, 20.0) + assert failures == [] + assert any("PASS" in line for line in lines) + + def test_tolerance_is_applied_before_failing(self) -> None: + # 10 000 ms × 2 × 1.20 = 24 000 ms ceiling. + assert smoke.evaluate_budgets({"full_response_ms": 23_999}, 2.0)[0] == [] + assert smoke.evaluate_budgets({"full_response_ms": 24_001}, 2.0)[0] + + def test_unbudgeted_metrics_are_not_checked(self) -> None: + # session_start_ms and debrief_ms are reported but have no budget: the + # opening is authored text and debrief latency has no documented SLO. + assert "session_start_ms" not in smoke.BUDGETS_MS + assert "debrief_ms" not in smoke.BUDGETS_MS + failures, lines = smoke.evaluate_budgets( + {"session_start_ms": 10 ** 9, "debrief_ms": 10 ** 9}, 1.0 + ) + assert failures == [] + assert lines == [] + + def test_missing_measurement_is_skipped_not_failed(self) -> None: + assert smoke.evaluate_budgets({}, 20.0) == ([], []) + + def test_failure_message_shows_the_arithmetic(self) -> None: + failures, _ = smoke.evaluate_budgets({"full_response_ms": 10 ** 6}, 20.0) + assert "documented budget 10000 ms × 20.0 × 1.2" in failures[0] + + +# --------------------------------------------------------------------------- +# Wall-clock budget +# --------------------------------------------------------------------------- + + +class TestDeadline: + def test_phase_durations_are_recorded(self) -> None: + clock = smoke.Deadline(60.0) + clock.enter("checksum") + clock.enter("runtime_start") + durations = clock.finish() + assert set(durations) >= {"startup", "checksum", "runtime_start"} + + def test_exhausted_budget_raises_timeout_naming_the_phase(self) -> None: + clock = smoke.Deadline(0.0) + with pytest.raises(smoke.SmokeFailure) as exc_info: + clock.enter("debrief") + assert exc_info.value.failure_class == smoke.FailureClass.TIMEOUT + assert exc_info.value.exit_code == 6 + # The phase recorded is the one that ran out of budget. + assert "debrief" in str(exc_info.value) + + def test_cap_clamps_a_request_timeout_to_the_remaining_budget(self) -> None: + clock = smoke.Deadline(30.0) + assert clock.cap(10.0) == 10.0 + assert clock.cap(10_000.0) <= 30.0 + + def test_cap_never_returns_a_useless_zero_timeout(self) -> None: + clock = smoke.Deadline(1.0) + assert clock.cap(600.0) >= 5.0 + + def test_finish_does_not_raise_on_an_exhausted_budget(self) -> None: + # finish() runs from run_smoke's finally block, where the timeout has + # already been classified; raising there would discard the verdict. + clock = smoke.Deadline(0.0) + assert "startup" in clock.finish() + + +# --------------------------------------------------------------------------- +# Reporting +# --------------------------------------------------------------------------- + + +class TestStepSummary: + def test_failure_summary_names_class_and_remedy(self) -> None: + text = smoke.render_step_summary({ + "verdict": "fail", + "model_id": "qwen3-4b", + "failure_class": smoke.FailureClass.RUNTIME, + "exit_code": 4, + "failures": ["convsim-core exited with code 1"], + "wall_clock_s": 123.0, + "wall_clock_budget_s": 1500.0, + "phase_durations_s": {"runtime_start": 42.0}, + }) + assert "FAIL" in text + assert "`runtime`" in text + assert "convsim-core exited with code 1" in text + assert smoke.REMEDIES[smoke.FailureClass.RUNTIME] in text + + def test_pass_summary_tabulates_measurements_with_ceilings(self) -> None: + text = smoke.render_step_summary({ + "verdict": "pass", + "model_id": "qwen3-4b", + "ci_hardware_factor": 20.0, + "measured_ms": {"full_response_ms": 110_000, "debrief_ms": 200_000}, + "wall_clock_s": 900.0, + "wall_clock_budget_s": 1500.0, + }) + assert "PASS" in text + assert "240000 ms" in text # 10 000 × 20 × 1.2 + assert "| `debrief_ms` | 200000 ms | — |" in text + + def test_summary_is_not_written_without_the_github_env_var( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.delenv("GITHUB_STEP_SUMMARY", raising=False) + smoke._write_step_summary({"verdict": "pass"}) # must not raise + + def test_summary_is_appended_when_github_env_var_is_set( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + path = tmp_path / "summary.md" + monkeypatch.setenv("GITHUB_STEP_SUMMARY", str(path)) + smoke._write_step_summary({"verdict": "pass", "model_id": "m"}) + assert "PASS" in path.read_text(encoding="utf-8") + + def test_report_is_valid_json(self, tmp_path: Path) -> None: + path = tmp_path / "nested" / "report.json" + smoke._write_report(path, {"verdict": "pass", "failures": []}) + assert json.loads(path.read_text(encoding="utf-8"))["verdict"] == "pass" + + +# --------------------------------------------------------------------------- +# Event-payload extraction +# --------------------------------------------------------------------------- + + +class TestEventExtraction: + def test_npc_turn_content_is_extracted(self) -> None: + events = [ + {"event_type": "player_turn", "payload": {"content": "hi"}}, + {"event_type": "npc_turn", "payload": {"content": "Go on."}}, + ] + assert smoke._npc_turn_content(events) == "Go on." + + def test_npc_opening_content_is_extracted(self) -> None: + events = [{"event_type": "npc_opening", "payload": {"content": "Welcome."}}] + assert smoke._npc_turn_content(events) == "Welcome." + + def test_absent_npc_event_yields_empty_string(self) -> None: + assert smoke._npc_turn_content([{"event_type": "tts_chunk", "payload": {}}]) == "" + + def test_excerpt_is_bounded_and_collapsed(self) -> None: + excerpt = smoke._excerpt("a\n\n b" + "x" * 500) + assert excerpt.startswith("a b") + assert len(excerpt) <= smoke.EXCERPT_CHARS + 1 # + the ellipsis + assert excerpt.endswith("…") + + +# --------------------------------------------------------------------------- +# Scripted conversation shape +# --------------------------------------------------------------------------- + + +class TestScriptedConversation: + def test_the_script_is_a_multi_turn_conversation(self) -> None: + assert len(smoke.SCRIPTED_PLAYER_TURNS) >= 2 + assert all(t.strip() for t in smoke.SCRIPTED_PLAYER_TURNS) + + def test_the_script_matches_the_fake_runtime_playthrough(self) -> None: + # Keeping the two harnesses on the same script means the nightly + # exercises the conversation shape the per-PR smoke already covers. + source = (REPO_ROOT / "tests" / "e2e" / "test_scripted_playthrough.py").read_text( + encoding="utf-8" + ) + for turn in smoke.SCRIPTED_PLAYER_TURNS: + assert turn in source, f"scripted turn drifted from the e2e playthrough: {turn!r}" + + +# --------------------------------------------------------------------------- +# Full-run orchestration, with llama-server and convsim-core faked out +# --------------------------------------------------------------------------- + + +class _FakeProc: + """Minimum of subprocess.Popen that run_smoke touches.""" + + def __init__(self, returncode: int | None = None) -> None: + self.stdout = iter(()) + self.stderr = iter(()) + self._returncode = returncode + self.terminated = False + + @property + def returncode(self) -> int | None: + return self._returncode + + def poll(self) -> int | None: + return self._returncode + + def terminate(self) -> None: + self.terminated = True + + def wait(self, timeout: float | None = None) -> int: + return self._returncode or 0 + + def kill(self) -> None: # pragma: no cover - only on a hung child + self.terminated = True + + +_OPENING = {"events": [{"event_type": "npc_opening", + "payload": {"content": "Thanks for coming in today."}}]} +_NPC_TURN = {"events": [{"event_type": "npc_turn", + "payload": {"content": "Walk me through that trade-off."}}], + "ending_type": None} + + +def _fake_core(debrief: dict, *, debug_turns: list | None = None): + """Build a _request_json stand-in that answers convsim-core's endpoints.""" + calls: list[str] = [] + + def _request_json(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + calls.append(url) + if url.endswith("/health"): + return {"llm_runtime": {"runtime_id": "llama_cpp"}} + if url.endswith("/api/sessions"): + return {"session_id": "sess-1"} + if url.endswith("/start"): + return _OPENING + if url.endswith("/turn"): + return _NPC_TURN + if url.endswith("/debug"): + return {"turns": debug_turns if debug_turns is not None else [ + {"turn_number": n, "used_fallback": False, + "used_native_structured_output": True} + for n in range(1, len(smoke.SCRIPTED_PLAYER_TURNS) + 1) + ]} + if url.endswith("/end"): + return {"state": "Ended"} + if url.endswith("/debrief"): + return debrief + raise AssertionError(f"unexpected request to {url}") + + _request_json.calls = calls # type: ignore[attr-defined] + return _request_json + + +@pytest.fixture() +def staged_model(tmp_path: Path) -> tuple[Path, str, str]: + """A fake GGUF on disk plus its id and digest.""" + models_dir = tmp_path / "models" + models_dir.mkdir() + payload = b"\x00gguf stand-in" + (models_dir / "test-model.gguf").write_bytes(payload) + return models_dir, "test-model", hashlib.sha256(payload).hexdigest() + + +@pytest.fixture() +def fake_servers(monkeypatch: pytest.MonkeyPatch) -> dict[str, _FakeProc]: + """Replace both child processes and the readiness polling with fakes.""" + procs = {"llama": _FakeProc(), "core": _FakeProc()} + monkeypatch.setattr(smoke, "_start_llama_server", lambda *a, **k: procs["llama"]) + monkeypatch.setattr(smoke, "_start_core", lambda *a, **k: procs["core"]) + monkeypatch.setattr(smoke, "_wait_for_http", lambda *a, **k: None) + return procs + + +class TestRunSmokeOrchestration: + """The phases, the report and the exit code, without a real model.""" + + def test_healthy_run_passes_and_reports( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core(_debrief())) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == 0 + results = json.loads(report.read_text(encoding="utf-8")) + assert results["verdict"] == "pass" + assert results["failure_class"] is None + assert results["runtime_id"] == "llama_cpp" + # One authored opening + every scripted player turn. + assert len(results["turns"]) == len(smoke.SCRIPTED_PLAYER_TURNS) + 1 + assert results["turns"][0]["model_generated"] is False + assert all(t["model_generated"] for t in results["turns"][1:]) + assert all(t["used_native_structured_output"] for t in results["turns"][1:]) + assert results["debrief"]["overall_score"] == 52.0 + assert set(results["measured_ms"]) >= { + "session_start_ms", "full_response_ms", "debrief_ms" + } + assert set(results["phase_durations_s"]) >= { + "checksum", "runtime_start", "conversation", "debrief", "assertions", "budget" + } + assert results["warnings"] == [] + + def test_both_children_are_stopped_even_on_a_pass( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch + ) -> None: + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core(_debrief())) + smoke.run_smoke(model_id, 20.0, None, model_sha256=digest, models_dir=models_dir) + assert fake_servers["llama"].terminated + assert fake_servers["core"].terminated + + def test_unscored_debrief_is_a_pipeline_failure( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, digest = staged_model + monkeypatch.setattr( + smoke, "_request_json", + _fake_core(_debrief(scores={}, overall_score=None)), + ) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failure_class"] == smoke.FailureClass.PIPELINE + assert results["failed_phase"] == "assertions" + assert any("rubric dimension scores" in f for f in results["failures"]) + + def test_latency_regression_is_a_budget_failure_not_a_pipeline_one( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + import time + + models_dir, model_id, digest = staged_model + base = _fake_core(_debrief()) + + def _slow_turn(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/turn"): + time.sleep(0.02) # measurable, so the 0 ms ceiling is exceeded + return base(url, payload=payload, timeout=timeout, expect=expect) + + monkeypatch.setattr(smoke, "_request_json", _slow_turn) + report = tmp_path / "report.json" + + # Factor 0 ⇒ a 0 ms ceiling, so any measurable latency regresses. + exit_code = smoke.run_smoke( + model_id, 0.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.BUDGET] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failure_class"] == smoke.FailureClass.BUDGET + assert results["failed_phase"] == "budget" + + def test_a_crashed_child_outranks_the_client_side_symptom( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # convsim-core dies mid-run: the client sees a pipeline-shaped error, but + # the real story is the crash, so the verdict must be `runtime`. + models_dir, model_id, digest = staged_model + fake_servers["core"]._returncode = 1 + + def _boom(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/health"): + return {"llm_runtime": {"runtime_id": "llama_cpp"}} + raise smoke.SmokeFailure(smoke.FailureClass.PIPELINE, "connection reset") + + monkeypatch.setattr(smoke, "_request_json", _boom) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.RUNTIME] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failure_class"] == smoke.FailureClass.RUNTIME + assert "convsim-core exited with code 1" in results["failures"][0] + + def test_a_fake_runtime_is_refused( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, digest = staged_model + + def _fake_runtime(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + assert url.endswith("/health"), "must not play a turn on the fake runtime" + return {"llm_runtime": {"runtime_id": "fake"}} + + monkeypatch.setattr(smoke, "_request_json", _fake_runtime) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + results = json.loads(report.read_text(encoding="utf-8")) + assert "runtime_id" in results["failures"][0] + + def test_checksum_drift_stops_the_run_before_any_server_starts( + self, staged_model, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, _ = staged_model + + def _never(*args: object, **kwargs: object): + raise AssertionError("must not load a model that failed verification") + + monkeypatch.setattr(smoke, "_start_llama_server", _never) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256="0" * 64, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.CHECKSUM] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failed_phase"] == "checksum" + + def test_run_without_a_checksum_warns_that_drift_was_not_checked( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, _ = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core(_debrief())) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke(model_id, 20.0, report, models_dir=models_dir) + + assert exit_code == 0 + results = json.loads(report.read_text(encoding="utf-8")) + assert any("checksum drift was not checked" in w for w in results["warnings"]) + + def test_an_absent_model_is_a_download_failure( + self, fake_servers, tmp_path: Path + ) -> None: + exit_code = smoke.run_smoke( + "absent", 20.0, None, models_dir=tmp_path / "empty" + ) + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.DOWNLOAD] + + def test_exhausted_wall_clock_budget_is_a_timeout( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core(_debrief())) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir, + wall_clock_budget_s=0.0, + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.TIMEOUT] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failure_class"] == smoke.FailureClass.TIMEOUT + + def test_an_unexpected_harness_bug_is_reported_not_raised( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, digest = staged_model + + def _bug(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/health"): + return {"llm_runtime": {"runtime_id": "llama_cpp"}} + raise TypeError("harness bug") + + monkeypatch.setattr(smoke, "_request_json", _bug) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code != 0 + results = json.loads(report.read_text(encoding="utf-8")) + assert "bug in the smoke harness" in results["failures"][0] + + def test_a_scenario_that_ends_early_stops_the_scripted_turns( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, digest = staged_model + base = _fake_core(_debrief(), debug_turns=[ + {"turn_number": 1, "used_fallback": False, "used_native_structured_output": True}, + ]) + + def _ends_after_one(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/turn"): + return {**_NPC_TURN, "ending_type": "success"} + return base(url, payload=payload, timeout=timeout, expect=expect) + + monkeypatch.setattr(smoke, "_request_json", _ends_after_one) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == 0 + results = json.loads(report.read_text(encoding="utf-8")) + assert len(results["turns"]) == 2 # opening + the one turn that ended it From 25258d5e235a92070b43e5327e1cf3f06b69b205 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 01:39:29 -0400 Subject: [PATCH 02/58] Wire the nightly workflow to the new smoke phases and hold it to a 30 min budget MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The nightly job now drives the extended harness and documents what it costs: - timeout-minutes drops from 60 to 30, the budget issue #457 asks for, with the per-phase breakdown recorded in the workflow header (~16 min warm cache, ~18 min cold). The harness's own 25 min deadline trips first so a slow run names its phase instead of surfacing as "The operation was canceled". - actions/cache splits into restore + save, with the save gated on success(). actions/cache's post step saves even when the job failed, which would have written a GGUF that failed verification into the cache — under a key derived from the expected hash, so every later run would restore the same bad bytes. - The run re-verifies the model against the registry checksum, so a drifted cache hit fails with exit 3 rather than being loaded. - MODEL_CACHE_PREFIX gives the one lever that actually clears a poisoned cache entry, since Actions cache entries are immutable. - The inline Python that read registry.yaml is replaced by the script's --print-registry-model, so the cache key, URL and checksum share one lookup. The harness's own unit tests run per-PR in the acceptance job. They need no model and take under a second, and without them the harness is only ever exercised at 04:00 UTC — so a typo in it shows up as a mysteriously red nightly rather than a red PR. Verified with actionlint across every workflow. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 9 ++ .github/workflows/model-smoke-nightly.yml | 137 ++++++++++++++-------- 2 files changed, 100 insertions(+), 46 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 53500258..f6a622bf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -231,6 +231,15 @@ jobs: run: | echo "Local: python -m pytest tests/acceptance/test_network_guard.py -v" python -m pytest tests/acceptance/test_network_guard.py -v + # The real-model nightly (model-smoke-nightly.yml) cannot run per-PR, but + # its decision logic can: checksum verification, the end-to-end + # assertions, budget arithmetic and failure classification. Without this + # the harness is only ever exercised at 04:00 UTC, so a typo in it shows up + # as a mysteriously red nightly. + - name: Run nightly real-model smoke harness unit tests (no model needed) + run: | + echo "Local: python -m pytest tests/scripts/ -v" + python -m pytest tests/scripts/ -v desktop-check: name: Desktop Rust check diff --git a/.github/workflows/model-smoke-nightly.yml b/.github/workflows/model-smoke-nightly.yml index bdb9ab99..811a9b75 100644 --- a/.github/workflows/model-smoke-nightly.yml +++ b/.github/workflows/model-smoke-nightly.yml @@ -1,23 +1,57 @@ # SPDX-License-Identifier: Apache-2.0 -# Nightly real-model smoke test. +# Nightly real-model smoke test (issue #457). # -# Downloads the pinned starter model from the registry, starts a local -# llama-server (llama-cpp-python's OpenAI-compatible server) plus convsim-core -# wired to it, runs a scripted end-to-end session through convsim-core's REST -# API, measures full-response latency, and fails if it regresses more than -# 20 % above the documented budget. +# Nightly, NOT per-PR: this is the only CI job that plays a conversation on a +# real local model. Every other job validates against the fake/scripted +# runtimes, so a regression that only shows up under real inference — a prompt +# that the model cannot satisfy, a grammar the adapter mis-serialises, a +# debrief that never scores — is invisible until this job runs. # -# convsim-core's turn endpoint is synchronous (token streaming lives in the -# apps/api gateway, which is not stood up here), so the smoke measures full -# end-to-end turn latency against the FULL_RESPONSE_MS budget (10 s). +# What it does: download the smallest registry model (role `starter`, Qwen3 4B +# Q4_K_M, ~2.5 GB) with SHA-256 verification, start llama-cpp-python's +# OpenAI-compatible server on it, start convsim-core wired to that server +# (CONVSIM_RUNTIME_ID=llama_cpp), play a scripted multi-turn conversation +# through convsim-core's REST API, and assert a SCORED debrief comes back. +# Latency is checked against the documented budgets. Full detail, including +# the local repro command: docs/real-model-smoke.md. # -# The model file (~2.5 GB) is cached by its registry SHA-256 so repeated -# nightly runs skip the download. The first run after a pinned-URL change -# re-downloads and re-caches the new file. +# ── Failure classification ─────────────────────────────────────────────────── +# scripts/nightly-model-smoke.py attributes every failure to one class and +# exits with a distinct code, so a red nightly is triageable from the job +# summary without reading logs: # -# Timings are measured on GitHub-hosted ubuntu-latest (2 vCPU, ~8 GB RAM, -# CPU-only inference). The CI budget is the documented budget multiplied by -# a hardware-tier factor that accounts for the slower runner: +# 1 budget — pipeline worked, latency regressed +# 2 download — model could not be fetched (network/HTTP/empty cache) +# 3 checksum — SHA-256 drift against the registry +# 4 runtime — llama-server or convsim-core crashed, hung, or 5xx'd +# 5 pipeline — servers healthy, end-to-end assertion failed +# 6 timeout — wall-clock budget exhausted (failing phase named) +# +# The script writes the class, the failures, the remedy, per-phase durations +# and measured latencies to $GITHUB_STEP_SUMMARY, and the full JSON report is +# uploaded as an artifact. +# +# ── Runtime budget: target < 30 min on standard GitHub runners ─────────────── +# timeout-minutes: 30 is the hard ceiling; the script's own +# --wall-clock-budget-s 1500 (25 min) trips first so the failure is attributed +# to a phase instead of surfacing as an unattributable job cancellation. +# Observed breakdown on ubuntu-latest (CPU-only): +# +# pip install (cached wheels) ~2 min +# model download (cache miss only) ~2 min — ~0 on a cache hit +# SHA-256 verification (2.5 GB, ×2 cold) ~1 min — ~0.5 min on a cache hit +# llama-server model load ~0.5 min +# 3 scripted turns + authored opening ~7 min +# debrief generation ~4 min +# ------------------------------------------------ +# total ~16 min warm cache, ~18 min cold +# +# If this creeps past ~25 min, reduce SCRIPTED_PLAYER_TURNS rather than raising +# the ceiling: a nightly that routinely runs near its timeout flaps. +# +# ── Latency budget on CI hardware ──────────────────────────────────────────── +# Timings are measured on GitHub-hosted ubuntu-latest, CPU-only inference. +# The CI budget is the documented budget multiplied by a hardware-tier factor: # # CI budget = documented budget × CI_HARDWARE_FACTOR (currently 20) # @@ -27,10 +61,11 @@ # 10 s documented budget. 20× gives a 10 s × 20 = 200 s CI budget and, with # the script's 1.20 regression tolerance, a 240 s ceiling — roughly 2× headroom # over the observed latency so normal runner-to-runner variance does not flap -# the nightly, while a genuine >2× regression still fails. This factor only -# models CI-hardware slowness; the product's 10 s target-hardware SLO is -# unchanged. Re-measure and re-tune if the runner class or starter model -# changes. +# the nightly, while a genuine >2× regression still fails. The headline +# full_response_ms is the MEDIAN of the scripted turns, so one unlucky turn +# cannot flap the job. This factor only models CI-hardware slowness; the +# product's 10 s target-hardware SLO is unchanged. Re-measure and re-tune if +# the runner class or starter model changes. name: Model smoke nightly @@ -42,11 +77,19 @@ on: permissions: contents: read +env: + # Bump this prefix to abandon a poisoned model cache entry (Actions cache + # entries are immutable, so a corrupt GGUF under a given key would otherwise + # be restored on every re-run — see the checksum remedy in the script). + MODEL_CACHE_PREFIX: model-gguf-v1 + jobs: model-smoke: name: Starter-model end-to-end smoke runs-on: ubuntu-latest - timeout-minutes: 60 + # Hard ceiling for the documented < 30 min runtime budget. The script's + # own 25 min wall-clock budget trips first and names the failing phase. + timeout-minutes: 30 steps: - uses: actions/checkout@v5 @@ -70,37 +113,26 @@ jobs: pip install "llama-cpp-python[server]" \ --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu - - name: Read starter model metadata from registry + # Single source of truth: the cache key, the download URL and the + # checksum the smoke verifies against all come from this one lookup, so + # they cannot drift apart. + - name: Resolve starter model from the registry id: registry run: | - python - <<'EOF' - import yaml, json, sys - from pathlib import Path - reg = yaml.safe_load(Path("model-registry/registry.yaml").read_text()) - starter = next(m for m in reg["models"] if m["role"] == "starter") - url = starter["download"]["url"] - sha256 = starter["download"]["sha256"] - model_id = starter["id"] - print(f"model_id={model_id}") - print(f"model_url={url}") - print(f"model_sha256={sha256}") - # Write to GITHUB_OUTPUT - import os - out = os.environ.get("GITHUB_OUTPUT", "/dev/stdout") - with open(out, "a") as f: - f.write(f"model_id={model_id}\n") - f.write(f"model_url={url}\n") - f.write(f"model_sha256={sha256}\n") - EOF - - - name: Cache model file + echo "Local: python scripts/nightly-model-smoke.py --print-registry-model starter" + python scripts/nightly-model-smoke.py --print-registry-model starter + + # Restore/save are split (rather than actions/cache) so a model that + # fails checksum verification is never written to the cache: the save + # step is gated on success(). + - name: Restore cached model file id: model-cache - uses: actions/cache@v5 + uses: actions/cache/restore@v5 with: path: ~/.convsim/models/llm - key: model-${{ steps.registry.outputs.model_sha256 }} + key: ${{ env.MODEL_CACHE_PREFIX }}-${{ steps.registry.outputs.model_sha256 }} - - name: Download starter model + - name: Download starter model (SHA-256 verified) if: steps.model-cache.outputs.cache-hit != 'true' run: | python scripts/nightly-model-smoke.py \ @@ -109,11 +141,24 @@ jobs: --model-sha256 "${{ steps.registry.outputs.model_sha256 }}" \ --model-id "${{ steps.registry.outputs.model_id }}" - - name: Run end-to-end smoke and check latency budgets + - name: Cache the verified model file + if: success() && steps.model-cache.outputs.cache-hit != 'true' + uses: actions/cache/save@v5 + with: + path: ~/.convsim/models/llm + key: ${{ env.MODEL_CACHE_PREFIX }}-${{ steps.registry.outputs.model_sha256 }} + + # --model-sha256 makes the run re-verify the file it is about to load, so + # a drifted or truncated CACHE HIT fails loudly (exit 3) instead of + # feeding corrupt weights to the smoke. + - name: Run real-model smoke (scripted conversation + scored debrief) run: | + echo "Local: see docs/real-model-smoke.md for the one-line local repro" python scripts/nightly-model-smoke.py \ --model-id "${{ steps.registry.outputs.model_id }}" \ + --model-sha256 "${{ steps.registry.outputs.model_sha256 }}" \ --ci-hardware-factor 20 \ + --wall-clock-budget-s 1500 \ --report-path /tmp/smoke-report.json - name: Upload smoke report From ce00999cc41e54211bdd7898f82b194d6b006618 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 01:39:37 -0400 Subject: [PATCH 03/58] Document the real-model smoke test: runtime budget, failure classes, local repro MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/real-model-smoke.md (mirrored to the docs site under Development) is the page a red nightly sends you to. It covers what the job proves, the failure class table with the first thing to do for each, the runtime budget with its measured per-phase breakdown and why the ceiling is 30 min, how the CI hardware factor is calibrated, what to do about checksum drift (and why never to "fix" it by updating the expected hash), and a copy-pasteable local repro. Also: ROADMAP marks the Milestone-1 polish item done, and both copies of the performance doc now link the nightly that actually enforces the latency budgets they state — the claim "nightly CI smoke tests flag any regression greater than 20 %" previously had no reachable referent. The offline-smoke-tests pages point at it too, since "real model verification" there means the network promise, not the end-to-end one. Closes #457 Co-Authored-By: Claude Opus 5 --- ROADMAP.md | 2 +- .../dev/adr-0001-npc-relationship-memory.md | 2 +- .../content/docs/dev/offline-smoke-tests.md | 4 + .../src/content/docs/dev/real-model-smoke.md | 225 ++++++++++++++++++ docs-site/src/content/docs/dev/screenshots.md | 2 +- .../src/content/docs/play/performance.md | 2 +- docs/offline-smoke-tests.md | 4 + docs/performance.md | 2 +- docs/real-model-smoke.md | 221 +++++++++++++++++ 9 files changed, 459 insertions(+), 5 deletions(-) create mode 100644 docs-site/src/content/docs/dev/real-model-smoke.md create mode 100644 docs/real-model-smoke.md diff --git a/ROADMAP.md b/ROADMAP.md index 6040c4e2..bea44234 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -91,7 +91,7 @@ criteria. - [ ] Real UI screenshots (replace SVG placeholders in README) — [#455](https://github.com/outrightmental/ConversationSimulator/issues/455) - [ ] Desktop app with bundled backend (Tauri sidecar for `convsim-core`) — [#456](https://github.com/outrightmental/ConversationSimulator/issues/456) -- [ ] Automated real-model CI smoke test — [#457](https://github.com/outrightmental/ConversationSimulator/issues/457) +- [x] Automated real-model CI smoke test — [#457](https://github.com/outrightmental/ConversationSimulator/issues/457) ### Post-alpha — Milestone 2+ diff --git a/docs-site/src/content/docs/dev/adr-0001-npc-relationship-memory.md b/docs-site/src/content/docs/dev/adr-0001-npc-relationship-memory.md index dd639cf2..9285d06d 100644 --- a/docs-site/src/content/docs/dev/adr-0001-npc-relationship-memory.md +++ b/docs-site/src/content/docs/dev/adr-0001-npc-relationship-memory.md @@ -2,7 +2,7 @@ title: "ADR-0001: NPC relationship memory" description: "Spike go/no-go report on persistent NPC relationship memory, covering the prototype, evaluation results, safety and privacy reviews, and a conditional-go recommendation." sidebar: - order: 14 + order: 15 --- **Status:** Proposed (spike branch — not merged) diff --git a/docs-site/src/content/docs/dev/offline-smoke-tests.md b/docs-site/src/content/docs/dev/offline-smoke-tests.md index a1409ff0..6befab0f 100644 --- a/docs-site/src/content/docs/dev/offline-smoke-tests.md +++ b/docs-site/src/content/docs/dev/offline-smoke-tests.md @@ -131,6 +131,10 @@ Exit codes: `0` = pass, `1` = network violation or pack error, `3` = unexpected To verify that the installed llama.cpp or Ollama runtime makes no *unexpected* outbound calls, run the smoke test while the real runtime is active. +> This page covers the **network** promise only. For the nightly CI job that +> plays a scripted conversation on a real model end-to-end and asserts a scored +> debrief, see [Real-model smoke test](/dev/real-model-smoke/). + ### 1. Start the local runtime ```bash diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md new file mode 100644 index 00000000..51508a2c --- /dev/null +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -0,0 +1,225 @@ +--- +title: "Real-model smoke test (nightly)" +description: "The nightly CI job that plays a scripted conversation on a real local model end-to-end — registry download, llama.cpp, convsim-core, scored debrief — with classified failures and a documented runtime budget." +sidebar: + order: 13 +--- + +Every other CI job validates Conversation Simulator against the **fake** and +**scripted** runtimes — deterministic stand-ins that need no model download. +That keeps pull requests fast, but it means a whole class of regression is +invisible per-PR: a prompt a real model cannot satisfy, a JSON schema the +adapter mis-serialises, a debrief that generates prose but never a score. + +The nightly real-model smoke closes that gap. It plays a scripted conversation +on a **real local model** end-to-end — registry download → llama.cpp → +`convsim-core` → scored debrief — and fails with a classified verdict. + +| | | +|---|---| +| Workflow | [`.github/workflows/model-smoke-nightly.yml`](https://github.com/outrightmental/ConversationSimulator/blob/main/.github/workflows/model-smoke-nightly.yml) | +| Harness | [`scripts/nightly-model-smoke.py`](https://github.com/outrightmental/ConversationSimulator/blob/main/scripts/nightly-model-smoke.py) | +| Unit tests (per-PR) | `tests/scripts/test_nightly_model_smoke.py` | +| Schedule | 04:00 UTC daily, plus **Run workflow** (`workflow_dispatch`) | +| Runner | GitHub-hosted `ubuntu-latest`, CPU-only inference | +| Model | registry role `starter` — Qwen3 4B Instruct Q4\_K\_M, ~2.5 GB, Apache-2.0 | +| Runtime budget | **< 30 min** (`timeout-minutes: 30`); the harness self-limits to 25 min | + +> Nightly, not per-PR, on purpose: a 2.5 GB download plus ~11 min of CPU-only +> inference cannot sit in the PR path. The per-PR equivalent is the fake-runtime +> playthrough in `tests/e2e/test_scripted_playthrough.py`, which uses the *same +> scripted player turns* so the two cover the same conversation shape. + +--- + +## What the nightly proves + +1. The pinned starter model downloads and its bytes match the registry SHA-256. +2. `llama-server` loads the GGUF and `convsim-core` reports + `runtime_id == "llama_cpp"` — the smoke refuses to run on a fake runtime. +3. A real model drives a multi-turn conversation on the built-in + `behavioral_interview` scenario, producing NPC turns that satisfy + `turn-output.schema.json`. +4. The session ends and produces a **scored** debrief: at least one rubric + dimension scored, a numeric `overall_score` in `[0, 100]`, and a non-empty + summary. +5. End-to-end turn latency stays within the documented budget, scaled for CI + hardware. + +The NPC *opening* line is authored scenario text, not a generation — so the +scripted player turns, not the opening, are what prove the model is working. + +--- + +## Failure classification + +A red nightly must be triageable from the job summary alone, so every failure is +attributed to exactly one class, with its own exit code, its own banner in the +log, and a remedy printed next to it: + +| Exit | Class | Means | First thing to do | +|---|---|---|---| +| 1 | `budget` | Pipeline worked; latency regressed past the CI ceiling | Compare `measured_ms` in the report artifact against recent nightlies | +| 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | +| 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | +| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx | Read the child stderr tail printed above the banner | +| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact | +| 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | + +The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download or +checksum failure says nothing about the app, a runtime failure is a crash, and a +pipeline failure means the model ran and produced output the product rejected. + +Each run writes the verdict, failure class, remedy, measured latencies and +per-phase durations to the GitHub **step summary**, and uploads the full JSON +report as the `model-smoke-report` artifact (30-day retention). + +--- + +## Runtime budget + +Target: **under 30 minutes on a standard GitHub-hosted runner.** The job sets +`timeout-minutes: 30` as a hard ceiling, and the harness runs with +`--wall-clock-budget-s 1500` (25 min) so *it* fails first and names the phase +that ran long — a GitHub-side timeout would only say "the operation was +canceled". + +Measured on `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): + +| Phase | Cold cache | Warm cache | +|---|---|---| +| `pip install` (prompt-composer, convsim-core, llama-cpp-python wheel) | ~2 min | ~2 min | +| Model download (2.5 GB from Hugging Face) | ~2 min | — | +| SHA-256 verification | ~1 min (×2) | ~0.5 min | +| `llama-server` model load | ~0.5 min | ~0.5 min | +| Authored opening + 3 scripted turns | ~7 min | ~7 min | +| Debrief generation | ~4 min | ~4 min | +| **Total** | **~18 min** | **~16 min** | + +The model is cached between runs under the key +`model-gguf-v1-`, so the download only recurs when the registry +pin changes. Cache *restore* and *save* are separate steps and the save is gated +on `success()`, so a file that fails verification is never written to the cache. + +If the total creeps past ~25 min, shorten `SCRIPTED_PLAYER_TURNS` rather than +raising the ceiling: a nightly that routinely runs near its timeout flaps. + +### Latency budget on CI hardware + +CPU-only CI hardware is far slower than the mid-spec reference machine the +product budgets target ([Performance and hardware tiers](/play/performance/)), so the harness +scales the documented budget: + +``` +CI ceiling = documented budget × CI_HARDWARE_FACTOR × REGRESSION_TOLERANCE + = 10 000 ms × 20 × 1.20 = 240 000 ms +``` + +`CI_HARDWARE_FACTOR` (20) is calibrated empirically: a full behavioral-interview +turn on this runner measures ~116 s, i.e. ~11.6× the 10 s documented budget. The +factor leaves roughly 2× headroom so runner-to-runner variance does not flap the +nightly, while a genuine >2× regression still fails. The headline +`full_response_ms` is the **median** of the scripted turns, so one unlucky turn +cannot flap the job either. + +This factor models CI slowness only — the product's 10 s target-hardware budget +is unchanged. Re-measure and re-tune it if the runner class or the starter model +changes. + +`session_start_ms` and `debrief_ms` are measured and reported but **not** +budget-checked: the NPC opening is authored text rather than inference, and +debrief generation has no documented SLO. + +--- + +## Checksum drift + +The checksum is verified in two places: right after a fresh download, and again +at the start of every run (including cache hits) before the weights are loaded. +A cache hit with drifted bytes therefore fails loudly instead of silently +feeding a corrupt model to the smoke. + +Exit 3 means one of two things: + +- **The pinned upstream file was replaced.** All registry URLs are pinned to a + specific Hugging Face revision, so this should not happen silently — if it + did, review the new file and re-pin with `scripts/pin-model.py`. +- **The cached download is corrupt or truncated.** The harness deletes the bad + local file, but Actions cache entries are immutable, so a poisoned entry would + be restored again on the next run. Delete it + (`gh cache delete model-gguf-v1-`) or bump `MODEL_CACHE_PREFIX` in the + workflow. + +Never "fix" a drift by updating the expected hash to whatever is on disk — that +is exactly the check this job exists to perform. + +--- + +## Running it locally + +You need the starter model on disk and `llama-cpp-python[server]` installed. +Nothing here touches the network except the one-time model download. + +```bash +# 1. Install the harness dependencies +pip install -e "packages/prompt-composer[dev]" +pip install -e "services/convsim-core[dev]" +pip install "llama-cpp-python[server]" \ + --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu + +# 2. Resolve the starter model's id / url / sha256 from the registry +python scripts/nightly-model-smoke.py --print-registry-model starter + +# 3. Download it (~2.5 GB) into ~/.convsim/models/llm/, verifying the checksum +python scripts/nightly-model-smoke.py --download-only \ + --model-id qwen3-4b-instruct-q4_k_m \ + --model-url "" \ + --model-sha256 "" + +# 4. Run the smoke. --ci-hardware-factor 1 holds your machine to the real +# product budget; use 20 to reproduce what CI accepts. +python scripts/nightly-model-smoke.py \ + --model-id qwen3-4b-instruct-q4_k_m \ + --model-sha256 "" \ + --ci-hardware-factor 1 \ + --report-path /tmp/smoke-report.json +``` + +Useful extras: + +- `--verify-only` — re-verify an already-downloaded model and exit. +- `--models-dir ` — look for `.gguf` somewhere other than + `~/.convsim/models/llm/`. +- `--wall-clock-budget-s` — the self-imposed deadline (default 1500 s). + +The harness binds `llama-server` on port 7356 and `convsim-core` on port 7399; +stop anything already listening there first. + +To exercise the harness's own logic without a model: + +```bash +python -m pytest tests/scripts/ -v +``` + +--- + +## Privacy + +The conversation is 100 % scripted (`SCRIPTED_PLAYER_TURNS` in the harness) and +contains no user data, so short NPC excerpts are printed to the log and recorded +in the report artifact — they are the evidence that a real model, rather than a +canned fallback, drove the conversation. Sessions are created with +`save_transcript: false`, so no transcript file is written, and `convsim-core` +runs against a throwaway data directory that is deleted when the run ends. + +--- + +## Related + +- [Offline smoke tests](/dev/offline-smoke-tests/) — proves no cloud service is + contacted during play (fake runtime, per-PR). +- [Voice smoke tests](/dev/voice-smoke-tests/) — STT/TTS sidecar checks. +- [Performance and hardware tiers](/play/performance/) — where the latency budgets + come from. +- [Release checklist](/dev/release-checklist/) — Part F covers manual real-model + verification of a packaged build. diff --git a/docs-site/src/content/docs/dev/screenshots.md b/docs-site/src/content/docs/dev/screenshots.md index d14f168c..b34bfb31 100644 --- a/docs-site/src/content/docs/dev/screenshots.md +++ b/docs-site/src/content/docs/dev/screenshots.md @@ -2,7 +2,7 @@ title: "Screenshots & demo assets" description: "What the placeholder visual assets in docs/assets/ show, how they are licensed, and how to replace them with real screenshots at Milestone 1." sidebar: - order: 13 + order: 14 --- This document describes the visual assets in [`docs/assets/`](https://github.com/outrightmental/ConversationSimulator/tree/main/docs/assets) — what they show, where they diff --git a/docs-site/src/content/docs/play/performance.md b/docs-site/src/content/docs/play/performance.md index b654ecd0..3a443203 100644 --- a/docs-site/src/content/docs/play/performance.md +++ b/docs-site/src/content/docs/play/performance.md @@ -9,7 +9,7 @@ ConversationSimulator runs a local LLM, optionally local STT (Whisper) and TTS ( ## Latency budgets -These are the official latency budgets for the **mid-spec reference machine** (Apple M2 / NVIDIA RTX 3060 equivalent). In-app performance warnings fire when a measurement exceeds the corresponding budget. Nightly CI smoke tests flag any regression greater than 20 % against these values. +These are the official latency budgets for the **mid-spec reference machine** (Apple M2 / NVIDIA RTX 3060 equivalent). In-app performance warnings fire when a measurement exceeds the corresponding budget. The [nightly real-model smoke test](/dev/real-model-smoke/) flags any regression greater than 20 % against these values, scaled for CPU-only CI hardware. | Metric | Budget | Condition | |--------|--------|-----------| diff --git a/docs/offline-smoke-tests.md b/docs/offline-smoke-tests.md index 98c397b1..50b9c18b 100644 --- a/docs/offline-smoke-tests.md +++ b/docs/offline-smoke-tests.md @@ -127,6 +127,10 @@ Exit codes: `0` = pass, `1` = network violation or pack error, `3` = unexpected To verify that the installed llama.cpp or Ollama runtime makes no *unexpected* outbound calls, run the smoke test while the real runtime is active. +> This page covers the **network** promise only. For the nightly CI job that +> plays a scripted conversation on a real model end-to-end and asserts a scored +> debrief, see [Real-model smoke test](real-model-smoke.md). + ### 1. Start the local runtime ```bash diff --git a/docs/performance.md b/docs/performance.md index 798bd724..5b07f168 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -4,7 +4,7 @@ ConversationSimulator runs a local LLM, optionally local STT (Whisper) and TTS ( ## Latency budgets -These are the official latency budgets for the **mid-spec reference machine** (Apple M2 / NVIDIA RTX 3060 equivalent). In-app performance warnings fire when a measurement exceeds the corresponding budget. Nightly CI smoke tests flag any regression greater than 20 % against these values. +These are the official latency budgets for the **mid-spec reference machine** (Apple M2 / NVIDIA RTX 3060 equivalent). In-app performance warnings fire when a measurement exceeds the corresponding budget. The [nightly real-model smoke test](real-model-smoke.md) flags any regression greater than 20 % against these values, scaled for CPU-only CI hardware. | Metric | Budget | Condition | |--------|--------|-----------| diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md new file mode 100644 index 00000000..e2aa263e --- /dev/null +++ b/docs/real-model-smoke.md @@ -0,0 +1,221 @@ + +# Real-model smoke test (nightly) + +Every other CI job validates Conversation Simulator against the **fake** and +**scripted** runtimes — deterministic stand-ins that need no model download. +That keeps pull requests fast, but it means a whole class of regression is +invisible per-PR: a prompt a real model cannot satisfy, a JSON schema the +adapter mis-serialises, a debrief that generates prose but never a score. + +The nightly real-model smoke closes that gap. It plays a scripted conversation +on a **real local model** end-to-end — registry download → llama.cpp → +`convsim-core` → scored debrief — and fails with a classified verdict. + +| | | +|---|---| +| Workflow | [`.github/workflows/model-smoke-nightly.yml`](../.github/workflows/model-smoke-nightly.yml) | +| Harness | [`scripts/nightly-model-smoke.py`](../scripts/nightly-model-smoke.py) | +| Unit tests (per-PR) | `tests/scripts/test_nightly_model_smoke.py` | +| Schedule | 04:00 UTC daily, plus **Run workflow** (`workflow_dispatch`) | +| Runner | GitHub-hosted `ubuntu-latest`, CPU-only inference | +| Model | registry role `starter` — Qwen3 4B Instruct Q4\_K\_M, ~2.5 GB, Apache-2.0 | +| Runtime budget | **< 30 min** (`timeout-minutes: 30`); the harness self-limits to 25 min | + +> Nightly, not per-PR, on purpose: a 2.5 GB download plus ~11 min of CPU-only +> inference cannot sit in the PR path. The per-PR equivalent is the fake-runtime +> playthrough in `tests/e2e/test_scripted_playthrough.py`, which uses the *same +> scripted player turns* so the two cover the same conversation shape. + +--- + +## What the nightly proves + +1. The pinned starter model downloads and its bytes match the registry SHA-256. +2. `llama-server` loads the GGUF and `convsim-core` reports + `runtime_id == "llama_cpp"` — the smoke refuses to run on a fake runtime. +3. A real model drives a multi-turn conversation on the built-in + `behavioral_interview` scenario, producing NPC turns that satisfy + `turn-output.schema.json`. +4. The session ends and produces a **scored** debrief: at least one rubric + dimension scored, a numeric `overall_score` in `[0, 100]`, and a non-empty + summary. +5. End-to-end turn latency stays within the documented budget, scaled for CI + hardware. + +The NPC *opening* line is authored scenario text, not a generation — so the +scripted player turns, not the opening, are what prove the model is working. + +--- + +## Failure classification + +A red nightly must be triageable from the job summary alone, so every failure is +attributed to exactly one class, with its own exit code, its own banner in the +log, and a remedy printed next to it: + +| Exit | Class | Means | First thing to do | +|---|---|---|---| +| 1 | `budget` | Pipeline worked; latency regressed past the CI ceiling | Compare `measured_ms` in the report artifact against recent nightlies | +| 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | +| 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | +| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx | Read the child stderr tail printed above the banner | +| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact | +| 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | + +The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download or +checksum failure says nothing about the app, a runtime failure is a crash, and a +pipeline failure means the model ran and produced output the product rejected. + +Each run writes the verdict, failure class, remedy, measured latencies and +per-phase durations to the GitHub **step summary**, and uploads the full JSON +report as the `model-smoke-report` artifact (30-day retention). + +--- + +## Runtime budget + +Target: **under 30 minutes on a standard GitHub-hosted runner.** The job sets +`timeout-minutes: 30` as a hard ceiling, and the harness runs with +`--wall-clock-budget-s 1500` (25 min) so *it* fails first and names the phase +that ran long — a GitHub-side timeout would only say "the operation was +canceled". + +Measured on `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): + +| Phase | Cold cache | Warm cache | +|---|---|---| +| `pip install` (prompt-composer, convsim-core, llama-cpp-python wheel) | ~2 min | ~2 min | +| Model download (2.5 GB from Hugging Face) | ~2 min | — | +| SHA-256 verification | ~1 min (×2) | ~0.5 min | +| `llama-server` model load | ~0.5 min | ~0.5 min | +| Authored opening + 3 scripted turns | ~7 min | ~7 min | +| Debrief generation | ~4 min | ~4 min | +| **Total** | **~18 min** | **~16 min** | + +The model is cached between runs under the key +`model-gguf-v1-`, so the download only recurs when the registry +pin changes. Cache *restore* and *save* are separate steps and the save is gated +on `success()`, so a file that fails verification is never written to the cache. + +If the total creeps past ~25 min, shorten `SCRIPTED_PLAYER_TURNS` rather than +raising the ceiling: a nightly that routinely runs near its timeout flaps. + +### Latency budget on CI hardware + +CPU-only CI hardware is far slower than the mid-spec reference machine the +product budgets target ([docs/performance.md](performance.md)), so the harness +scales the documented budget: + +``` +CI ceiling = documented budget × CI_HARDWARE_FACTOR × REGRESSION_TOLERANCE + = 10 000 ms × 20 × 1.20 = 240 000 ms +``` + +`CI_HARDWARE_FACTOR` (20) is calibrated empirically: a full behavioral-interview +turn on this runner measures ~116 s, i.e. ~11.6× the 10 s documented budget. The +factor leaves roughly 2× headroom so runner-to-runner variance does not flap the +nightly, while a genuine >2× regression still fails. The headline +`full_response_ms` is the **median** of the scripted turns, so one unlucky turn +cannot flap the job either. + +This factor models CI slowness only — the product's 10 s target-hardware budget +is unchanged. Re-measure and re-tune it if the runner class or the starter model +changes. + +`session_start_ms` and `debrief_ms` are measured and reported but **not** +budget-checked: the NPC opening is authored text rather than inference, and +debrief generation has no documented SLO. + +--- + +## Checksum drift + +The checksum is verified in two places: right after a fresh download, and again +at the start of every run (including cache hits) before the weights are loaded. +A cache hit with drifted bytes therefore fails loudly instead of silently +feeding a corrupt model to the smoke. + +Exit 3 means one of two things: + +- **The pinned upstream file was replaced.** All registry URLs are pinned to a + specific Hugging Face revision, so this should not happen silently — if it + did, review the new file and re-pin with `scripts/pin-model.py`. +- **The cached download is corrupt or truncated.** The harness deletes the bad + local file, but Actions cache entries are immutable, so a poisoned entry would + be restored again on the next run. Delete it + (`gh cache delete model-gguf-v1-`) or bump `MODEL_CACHE_PREFIX` in the + workflow. + +Never "fix" a drift by updating the expected hash to whatever is on disk — that +is exactly the check this job exists to perform. + +--- + +## Running it locally + +You need the starter model on disk and `llama-cpp-python[server]` installed. +Nothing here touches the network except the one-time model download. + +```bash +# 1. Install the harness dependencies +pip install -e "packages/prompt-composer[dev]" +pip install -e "services/convsim-core[dev]" +pip install "llama-cpp-python[server]" \ + --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu + +# 2. Resolve the starter model's id / url / sha256 from the registry +python scripts/nightly-model-smoke.py --print-registry-model starter + +# 3. Download it (~2.5 GB) into ~/.convsim/models/llm/, verifying the checksum +python scripts/nightly-model-smoke.py --download-only \ + --model-id qwen3-4b-instruct-q4_k_m \ + --model-url "" \ + --model-sha256 "" + +# 4. Run the smoke. --ci-hardware-factor 1 holds your machine to the real +# product budget; use 20 to reproduce what CI accepts. +python scripts/nightly-model-smoke.py \ + --model-id qwen3-4b-instruct-q4_k_m \ + --model-sha256 "" \ + --ci-hardware-factor 1 \ + --report-path /tmp/smoke-report.json +``` + +Useful extras: + +- `--verify-only` — re-verify an already-downloaded model and exit. +- `--models-dir ` — look for `.gguf` somewhere other than + `~/.convsim/models/llm/`. +- `--wall-clock-budget-s` — the self-imposed deadline (default 1500 s). + +The harness binds `llama-server` on port 7356 and `convsim-core` on port 7399; +stop anything already listening there first. + +To exercise the harness's own logic without a model: + +```bash +python -m pytest tests/scripts/ -v +``` + +--- + +## Privacy + +The conversation is 100 % scripted (`SCRIPTED_PLAYER_TURNS` in the harness) and +contains no user data, so short NPC excerpts are printed to the log and recorded +in the report artifact — they are the evidence that a real model, rather than a +canned fallback, drove the conversation. Sessions are created with +`save_transcript: false`, so no transcript file is written, and `convsim-core` +runs against a throwaway data directory that is deleted when the run ends. + +--- + +## Related + +- [Offline smoke tests](offline-smoke-tests.md) — proves no cloud service is + contacted during play (fake runtime, per-PR). +- [Voice smoke tests](voice-smoke-tests.md) — STT/TTS sidecar checks. +- [Performance and hardware tiers](performance.md) — where the latency budgets + come from. +- [Release checklist](release-checklist.md) — Part F covers manual real-model + verification of a packaged build. From 7b2e93de3112f23fb59c2dcbfd637e2b1838a971 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 02:11:59 -0400 Subject: [PATCH 04/58] Keep the nightly's verdict honest when the run does not go to plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-review of #507 found three paths where the smoke reports a class that is not what actually happened — the one thing #457 asks it to get right. Ending the session twice. The turn pipeline sets flow_state to 'Ended' as soon as a turn carries an ending_type (turn_pipeline.py), so the explicit POST /end that follows answers 409, and _request_json files 4xx as `pipeline`. A scenario that ended on its own terms would have failed the nightly with exit 5 and "an end-to-end assertion failed". Skip /end when the scenario already ended; the existing early-ending test faked /end as 200 and so could not see this, and now asserts the call is not made. A slow debrief read as a crashed server. CONVSIM_LLAMA_CPP_TIMEOUT was sized for one NPC turn (300 s against a measured ~116 s), but the debrief is the slowest single generation at ~240 s — 25 % headroom, against the ~2x this job holds everything else to. Past that the adapter times out, convsim-core answers 500, and slowness is reported as `runtime` with a remedy about runner OOM. Size the adapter ceiling for the debrief instead (DEBRIEF_SLOWDOWN_FACTOR x the full-response CI ceiling) and keep our own POST timeout above it, so the component that gives up first is the one whose error names the knob. A lost deadline. cap() shrinks each request timeout to the budget that remains, so the last call before the deadline dies client-side and raises RUNTIME — exit 4 for what is exactly the condition exit 6 exists to report. Reclassify as `timeout` when the clock is exhausted and no child crashed; a crashed child still outranks both. Also: --wall-clock-budget-s 1500 (25 min) could not reliably trip before the 30 min job timeout it is supposed to pre-empt, because timeout-minutes covers the whole job while the harness clock starts at its own step — checkout, three pip installs, the cache restore and a cold 2.5 GB download burn up to ~8 min the harness never sees. On exactly the cold-cache nights where an attributed verdict matters most, GitHub would cancel the job first. Drop it to 1200 s (20 min) and add a test that parses the workflow so the two budgets cannot drift back together. Co-Authored-By: Claude Opus 5 --- .github/workflows/model-smoke-nightly.yml | 15 +- .../src/content/docs/dev/real-model-smoke.md | 28 ++- docs/real-model-smoke.md | 28 ++- scripts/nightly-model-smoke.py | 92 +++++++--- tests/scripts/test_nightly_model_smoke.py | 169 ++++++++++++++++++ 5 files changed, 294 insertions(+), 38 deletions(-) diff --git a/.github/workflows/model-smoke-nightly.yml b/.github/workflows/model-smoke-nightly.yml index 811a9b75..f142059b 100644 --- a/.github/workflows/model-smoke-nightly.yml +++ b/.github/workflows/model-smoke-nightly.yml @@ -33,8 +33,10 @@ # # ── Runtime budget: target < 30 min on standard GitHub runners ─────────────── # timeout-minutes: 30 is the hard ceiling; the script's own -# --wall-clock-budget-s 1500 (25 min) trips first so the failure is attributed -# to a phase instead of surfacing as an unattributable job cancellation. +# --wall-clock-budget-s 1200 (20 min) trips first so the failure is attributed +# to a phase instead of surfacing as an unattributable job cancellation. The +# 10 min gap is not slack: the steps before the smoke (checkout, pip install, +# cache restore, a cold 2.5 GB download) run against the same job clock. # Observed breakdown on ubuntu-latest (CPU-only): # # pip install (cached wheels) ~2 min @@ -46,8 +48,9 @@ # ------------------------------------------------ # total ~16 min warm cache, ~18 min cold # -# If this creeps past ~25 min, reduce SCRIPTED_PLAYER_TURNS rather than raising -# the ceiling: a nightly that routinely runs near its timeout flaps. +# If the smoke step creeps past ~15 min — or the job past ~25 min — reduce +# SCRIPTED_PLAYER_TURNS rather than raising either ceiling: a nightly that +# routinely runs near its timeout flaps. # # ── Latency budget on CI hardware ──────────────────────────────────────────── # Timings are measured on GitHub-hosted ubuntu-latest, CPU-only inference. @@ -88,7 +91,7 @@ jobs: name: Starter-model end-to-end smoke runs-on: ubuntu-latest # Hard ceiling for the documented < 30 min runtime budget. The script's - # own 25 min wall-clock budget trips first and names the failing phase. + # own 20 min wall-clock budget trips first and names the failing phase. timeout-minutes: 30 steps: @@ -158,7 +161,7 @@ jobs: --model-id "${{ steps.registry.outputs.model_id }}" \ --model-sha256 "${{ steps.registry.outputs.model_sha256 }}" \ --ci-hardware-factor 20 \ - --wall-clock-budget-s 1500 \ + --wall-clock-budget-s 1200 \ --report-path /tmp/smoke-report.json - name: Upload smoke report diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 51508a2c..0ba6f2df 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -23,7 +23,7 @@ on a **real local model** end-to-end — registry download → llama.cpp → | Schedule | 04:00 UTC daily, plus **Run workflow** (`workflow_dispatch`) | | Runner | GitHub-hosted `ubuntu-latest`, CPU-only inference | | Model | registry role `starter` — Qwen3 4B Instruct Q4\_K\_M, ~2.5 GB, Apache-2.0 | -| Runtime budget | **< 30 min** (`timeout-minutes: 30`); the harness self-limits to 25 min | +| Runtime budget | **< 30 min** (`timeout-minutes: 30`); the harness self-limits to 20 min | > Nightly, not per-PR, on purpose: a 2.5 GB download plus ~11 min of CPU-only > inference cannot sit in the PR path. The per-PR equivalent is the fake-runtime @@ -70,6 +70,16 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. +When more than one class could apply, the harness reports the strongest evidence +rather than the symptom the client happened to see: + +1. **A crashed child wins.** A dead `convsim-core` reads as `runtime`, not as the + connection reset it caused downstream. +2. **An exhausted wall clock beats a transport error.** Each request timeout is + capped to the budget that remains, so the last call before the deadline dies + client-side and *looks* like an unresponsive server. It is reported as + `timeout`, whose remedy points at `phase_durations_s`. + Each run writes the verdict, failure class, remedy, measured latencies and per-phase durations to the GitHub **step summary**, and uploads the full JSON report as the `model-smoke-report` artifact (30-day retention). @@ -80,10 +90,17 @@ report as the `model-smoke-report` artifact (30-day retention). Target: **under 30 minutes on a standard GitHub-hosted runner.** The job sets `timeout-minutes: 30` as a hard ceiling, and the harness runs with -`--wall-clock-budget-s 1500` (25 min) so *it* fails first and names the phase +`--wall-clock-budget-s 1200` (20 min) so *it* fails first and names the phase that ran long — a GitHub-side timeout would only say "the operation was canceled". +The 10-minute gap between the two is not slack. `timeout-minutes` covers the +whole job, and the harness only starts after checkout, three `pip install` +steps, the cache restore and — on a cache miss — a 2.5 GB download: up to +~8 min that the harness's own clock never sees. A 25 min harness budget would +lose the race to the job timeout on exactly the cold-cache nights where an +attributed verdict matters most. + Measured on `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | @@ -101,8 +118,9 @@ The model is cached between runs under the key pin changes. Cache *restore* and *save* are separate steps and the save is gated on `success()`, so a file that fails verification is never written to the cache. -If the total creeps past ~25 min, shorten `SCRIPTED_PLAYER_TURNS` rather than -raising the ceiling: a nightly that routinely runs near its timeout flaps. +If the smoke step creeps past ~15 min — or the job past ~25 min — shorten +`SCRIPTED_PLAYER_TURNS` rather than raising either ceiling: a nightly that +routinely runs near its timeout flaps. ### Latency budget on CI hardware @@ -190,7 +208,7 @@ Useful extras: - `--verify-only` — re-verify an already-downloaded model and exit. - `--models-dir ` — look for `.gguf` somewhere other than `~/.convsim/models/llm/`. -- `--wall-clock-budget-s` — the self-imposed deadline (default 1500 s). +- `--wall-clock-budget-s` — the self-imposed deadline (default 1200 s). The harness binds `llama-server` on port 7356 and `convsim-core` on port 7399; stop anything already listening there first. diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index e2aa263e..543f97af 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -19,7 +19,7 @@ on a **real local model** end-to-end — registry download → llama.cpp → | Schedule | 04:00 UTC daily, plus **Run workflow** (`workflow_dispatch`) | | Runner | GitHub-hosted `ubuntu-latest`, CPU-only inference | | Model | registry role `starter` — Qwen3 4B Instruct Q4\_K\_M, ~2.5 GB, Apache-2.0 | -| Runtime budget | **< 30 min** (`timeout-minutes: 30`); the harness self-limits to 25 min | +| Runtime budget | **< 30 min** (`timeout-minutes: 30`); the harness self-limits to 20 min | > Nightly, not per-PR, on purpose: a 2.5 GB download plus ~11 min of CPU-only > inference cannot sit in the PR path. The per-PR equivalent is the fake-runtime @@ -66,6 +66,16 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. +When more than one class could apply, the harness reports the strongest evidence +rather than the symptom the client happened to see: + +1. **A crashed child wins.** A dead `convsim-core` reads as `runtime`, not as the + connection reset it caused downstream. +2. **An exhausted wall clock beats a transport error.** Each request timeout is + capped to the budget that remains, so the last call before the deadline dies + client-side and *looks* like an unresponsive server. It is reported as + `timeout`, whose remedy points at `phase_durations_s`. + Each run writes the verdict, failure class, remedy, measured latencies and per-phase durations to the GitHub **step summary**, and uploads the full JSON report as the `model-smoke-report` artifact (30-day retention). @@ -76,10 +86,17 @@ report as the `model-smoke-report` artifact (30-day retention). Target: **under 30 minutes on a standard GitHub-hosted runner.** The job sets `timeout-minutes: 30` as a hard ceiling, and the harness runs with -`--wall-clock-budget-s 1500` (25 min) so *it* fails first and names the phase +`--wall-clock-budget-s 1200` (20 min) so *it* fails first and names the phase that ran long — a GitHub-side timeout would only say "the operation was canceled". +The 10-minute gap between the two is not slack. `timeout-minutes` covers the +whole job, and the harness only starts after checkout, three `pip install` +steps, the cache restore and — on a cache miss — a 2.5 GB download: up to +~8 min that the harness's own clock never sees. A 25 min harness budget would +lose the race to the job timeout on exactly the cold-cache nights where an +attributed verdict matters most. + Measured on `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | @@ -97,8 +114,9 @@ The model is cached between runs under the key pin changes. Cache *restore* and *save* are separate steps and the save is gated on `success()`, so a file that fails verification is never written to the cache. -If the total creeps past ~25 min, shorten `SCRIPTED_PLAYER_TURNS` rather than -raising the ceiling: a nightly that routinely runs near its timeout flaps. +If the smoke step creeps past ~15 min — or the job past ~25 min — shorten +`SCRIPTED_PLAYER_TURNS` rather than raising either ceiling: a nightly that +routinely runs near its timeout flaps. ### Latency budget on CI hardware @@ -186,7 +204,7 @@ Useful extras: - `--verify-only` — re-verify an already-downloaded model and exit. - `--models-dir ` — look for `.gguf` somewhere other than `~/.convsim/models/llm/`. -- `--wall-clock-budget-s` — the self-imposed deadline (default 1500 s). +- `--wall-clock-budget-s` — the self-imposed deadline (default 1200 s). The harness binds `llama-server` on port 7356 and `convsim-core` on port 7399; stop anything already listening there first. diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 42dbd898..edb4c72a 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -76,7 +76,7 @@ class exit meaning --model-id qwen3-4b-instruct-q4_k_m \\ --model-sha256 \\ --ci-hardware-factor 20 \\ - --wall-clock-budget-s 1500 \\ + --wall-clock-budget-s 1200 \\ --report-path /tmp/smoke-report.json """ @@ -125,11 +125,15 @@ class exit meaning LLAMA_SERVER_PORT = 7356 # convsim-core's default CONVSIM_LLAMA_CPP_BASE_URL port CORE_PORT = 7399 -# Wall-clock budget for the whole run, in seconds. Deliberately below the -# workflow's timeout-minutes so the script — which knows which phase it is in — -# fails first with an attributed verdict, instead of GitHub killing the job -# with an unattributable "The operation was canceled". See docs/real-model-smoke.md. -DEFAULT_WALL_CLOCK_BUDGET_S = 1500.0 # 25 min +# Wall-clock budget for the whole run, in seconds. The script — which knows +# which phase it is in — must fail first with an attributed verdict instead of +# GitHub killing the job with an unattributable "The operation was canceled", +# so this has to clear the job's timeout-minutes *minus the steps that run +# before the script does*: checkout, pip install, cache restore and (on a cache +# miss) the 2.5 GB download, together up to ~8 min of the 30 min job. 20 min +# therefore leaves the script's own deadline the first one to trip. +# See docs/real-model-smoke.md. +DEFAULT_WALL_CLOCK_BUDGET_S = 1200.0 # 20 min # Scripted player turns — deliberately the same script as the fake-runtime # playthrough in tests/e2e/test_scripted_playthrough.py, so the real-model @@ -148,6 +152,11 @@ class exit meaning # A debrief summary shorter than this is not a usable coaching summary. MIN_SUMMARY_CHARS = 20 +# How much slower the debrief generation is allowed to be than one NPC turn +# before we stop calling it slow and start calling it hung. Only used to size +# request timeouts (see run_smoke) — debrief latency itself is not budget-checked. +DEBRIEF_SLOWDOWN_FACTOR = 2.0 + # ── Failure classification ──────────────────────────────────────────────────── @@ -457,10 +466,10 @@ def _start_core(data_dir: Path, port: int, llama_port: int, llama_timeout_s: flo # Point the adapter at the llama-server started above. "CONVSIM_LLAMA_CPP_BASE_URL": f"http://127.0.0.1:{llama_port}", "CONVSIM_LLAMA_CPP_CONTEXT_LENGTH": "8192", - # The adapter's per-request timeout must outlast the CI ceiling we - # measure against, or a slow-but-passing turn on the 2-vCPU CPU-only - # runner is killed by the adapter's 30 s default before we can time it, - # failing the nightly spuriously instead of reporting the real latency. + # The adapter's per-request timeout must outlast the slowest call we + # intend to measure (the debrief), or the adapter's 30 s default kills a + # slow-but-reportable generation first and convsim-core answers 500 — a + # latency problem misfiled as a RUNTIME crash. See run_smoke. "CONVSIM_LLAMA_CPP_TIMEOUT": str(int(llama_timeout_s)), "CONVSIM_DATA_DIR": str(data_dir), "CONVSIM_LOG_DIR": str(data_dir / "logs"), @@ -807,14 +816,24 @@ def run_smoke( ) -> int: """Run the full scripted smoke. Returns the process exit code.""" clock = Deadline(wall_clock_budget_s) - # The turn must be allowed to run past the CI ceiling before we judge it, so - # both the adapter timeout and our own POST timeout are derived from the - # full-response ceiling with headroom rather than a fixed 30 s / 180 s. + # A call must be allowed to run past the latency we are about to judge it + # on, or whoever gives up first decides the verdict instead of us: the + # adapter's own timeout surfaces as HTTP 500 (class `runtime`, "the server + # crashed") and our POST timeout surfaces as a transport error (also + # `runtime`), when the truth is a slow model — class `budget`. + # + # The debrief, not a turn, is the slowest single generation (one long + # structured document against one NPC reply), so the adapter ceiling is + # sized for it: DEBRIEF_SLOWDOWN_FACTOR × the full-response CI ceiling. + # Our own POST timeout then sits above the adapter's, so when something + # really does hang it is the adapter that reports it, with a message that + # names the knob. The run's wall-clock deadline — not these timeouts — is + # what bounds the job. full_ceiling_s = ( BUDGETS_MS["full_response_ms"] * ci_hardware_factor * REGRESSION_TOLERANCE ) / 1000 - llama_timeout_s = max(180.0, full_ceiling_s + 60.0) - turn_timeout_s = llama_timeout_s + 30.0 + llama_timeout_s = max(300.0, full_ceiling_s * DEBRIEF_SLOWDOWN_FACTOR + 60.0) + request_timeout_s = llama_timeout_s + 30.0 results: Dict[str, Any] = { "model_id": model_id, @@ -942,6 +961,7 @@ def run_smoke( }) turn_latencies: List[float] = [] + ended_by_scenario = False for i, player_text in enumerate(SCRIPTED_PLAYER_TURNS, start=1): clock.check() print(f"[smoke] Player turn {i}/{len(SCRIPTED_PLAYER_TURNS)}…") @@ -949,7 +969,7 @@ def run_smoke( turn = _request_json( f"{base}/sessions/{session_id}/turn", payload={"content": player_text}, - timeout=clock.cap(turn_timeout_s), + timeout=clock.cap(request_timeout_s), ) latency_ms = (time.monotonic() - t0) * 1000 turn_latencies.append(latency_ms) @@ -964,6 +984,10 @@ def run_smoke( "ending_type": turn.get("ending_type"), }) if turn.get("ending_type"): + # The turn pipeline already moved the session to 'Ended', so + # POST /end would 409 — see end_session() in the sessions + # router. Record that and skip the explicit end below. + ended_by_scenario = True print(f"[smoke] Scenario ended early (ending_type={turn['ending_type']}); " "stopping the scripted turns.") break @@ -987,14 +1011,19 @@ def run_smoke( # ── debrief ─────────────────────────────────────────────────────── clock.enter("debrief") - print("[smoke] Ending session…") - _request_json(f"{base}/sessions/{session_id}/end", payload={}, timeout=clock.cap(30.0)) + if ended_by_scenario: + print("[smoke] Session already Ended by the scenario; skipping /end.") + else: + print("[smoke] Ending session…") + _request_json( + f"{base}/sessions/{session_id}/end", payload={}, timeout=clock.cap(30.0) + ) print("[smoke] Generating debrief (real model call)…") t0 = time.monotonic() debrief = _request_json( f"{base}/sessions/{session_id}/debrief", payload={}, - timeout=clock.cap(turn_timeout_s * 2), + timeout=clock.cap(request_timeout_s), ) debrief_ms = (time.monotonic() - t0) * 1000 results["measured_ms"]["debrief_ms"] = round(debrief_ms) @@ -1044,10 +1073,23 @@ def run_smoke( results["exit_code"] = 0 except SmokeFailure as exc: - # A crashed child outranks whatever the client saw: "core exited with - # code 1" is the real story, not the connection reset it caused. + # Precedence, strongest evidence first: + # 1. A crashed child outranks whatever the client saw — "core exited + # with code 1" is the real story, not the connection reset it caused. + # 2. An exhausted wall clock outranks the transport error it produced: + # cap() shrinks each request timeout to the budget that is left, so + # the last request before the deadline dies client-side and reports + # itself as RUNTIME ("the server never answered"). The honest + # verdict is TIMEOUT, which is the class whose remedy says to look + # at phase_durations_s. crashed = _crashed_child((("llama-server", llama_proc), ("convsim-core", core_proc))) - failure_class = FailureClass.RUNTIME if crashed else exc.failure_class + out_of_time = not crashed and clock.remaining_s <= 0 + if crashed: + failure_class = FailureClass.RUNTIME + elif out_of_time: + failure_class = FailureClass.TIMEOUT + else: + failure_class = exc.failure_class results["verdict"] = "fail" results["failure_class"] = failure_class results["exit_code"] = EXIT_CODES[failure_class] @@ -1056,6 +1098,12 @@ def run_smoke( results["failures"] = [str(exc)] if crashed: results["failures"].insert(0, crashed) + elif out_of_time and exc.failure_class != FailureClass.TIMEOUT: + results["failures"].insert( + 0, + f"Wall-clock budget of {wall_clock_budget_s:.0f} s exhausted during phase " + f"{results['failed_phase']!r} (elapsed {clock.elapsed_s:.0f} s)", + ) _dump_stderr_tails((("convsim-core", core_stderr_tail), ("llama-server", llama_stderr_tail))) _print_banner(failure_class, results["failures"]) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 8b86616b..c6ac801c 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -358,6 +358,56 @@ def test_failure_message_shows_the_arithmetic(self) -> None: # --------------------------------------------------------------------------- +# Steps that burn the job's clock before the harness starts and so are invisible +# to its own deadline: checkout, three pip installs, the cache restore and, on a +# cache miss, the 2.5 GB download. Measured at ~8 min cold; see the breakdown in +# docs/real-model-smoke.md. +PRE_SMOKE_JOB_MINUTES = 8 + +_WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "model-smoke-nightly.yml" + + +def _workflow_text() -> str: + return _WORKFLOW_PATH.read_text(encoding="utf-8") + + +def _workflow_budget_s() -> float: + import re + + match = re.search(r"--wall-clock-budget-s (\d+)", _workflow_text()) + assert match, "the smoke step must pass an explicit --wall-clock-budget-s" + return float(match.group(1)) + + +def _job_timeout_minutes() -> int: + import yaml + + return yaml.safe_load(_workflow_text())["jobs"]["model-smoke"]["timeout-minutes"] + + +class TestWorkflowBudgetAgreement: + """The harness deadline must beat the job timeout, or exit 6 is unreachable.""" + + def test_the_harness_deadline_trips_before_the_job_timeout(self) -> None: + # timeout-minutes covers the whole job; the harness clock starts only at + # its own step. If the two are set as if they measured the same thing, + # GitHub cancels the job first and the attributed timeout never prints. + harness_min = _workflow_budget_s() / 60 + job_min = _job_timeout_minutes() + assert harness_min + PRE_SMOKE_JOB_MINUTES <= job_min, ( + f"a {harness_min:.0f} min harness budget plus ~{PRE_SMOKE_JOB_MINUTES} min of " + f"setup exceeds the {job_min} min job timeout, so GitHub cancels the job " + "before the harness can report an attributed timeout" + ) + + def test_the_documented_target_is_under_thirty_minutes(self) -> None: + # The acceptance criterion in #457: "< 30 min on standard GitHub runners". + assert _job_timeout_minutes() <= 30 + + def test_the_default_matches_what_the_workflow_passes(self) -> None: + assert _workflow_budget_s() == smoke.DEFAULT_WALL_CLOCK_BUDGET_S + + class TestDeadline: def test_phase_durations_are_recorded(self) -> None: clock = smoke.Deadline(60.0) @@ -770,6 +820,119 @@ def test_exhausted_wall_clock_budget_is_a_timeout( results = json.loads(report.read_text(encoding="utf-8")) assert results["failure_class"] == smoke.FailureClass.TIMEOUT + def test_a_normal_run_still_ends_the_session_explicitly( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, digest = staged_model + fake = _fake_core(_debrief()) + monkeypatch.setattr(smoke, "_request_json", fake) + + assert smoke.run_smoke( + model_id, 20.0, None, model_sha256=digest, models_dir=models_dir + ) == 0 + assert any(url.endswith("/end") for url in fake.calls) + + def test_request_timeouts_outlast_the_adapter_so_slowness_is_not_a_crash( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch + ) -> None: + # Whoever gives up first decides the verdict. The adapter timeout + # handed to convsim-core has to clear the debrief (the slowest single + # generation), and our own POST timeout has to clear the adapter's, or + # a slow model is reported as a crashed server instead of a budget + # regression. + models_dir, model_id, digest = staged_model + adapter_timeout: dict[str, float] = {} + post_timeouts: list[float] = [] + + def _record_core(data_dir, port, llama_port, llama_timeout_s): + adapter_timeout["s"] = llama_timeout_s + return fake_servers["core"] + + base = _fake_core(_debrief()) + + def _record_timeout(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith(("/turn", "/debrief")): + post_timeouts.append(timeout) + return base(url, payload=payload, timeout=timeout, expect=expect) + + monkeypatch.setattr(smoke, "_start_core", _record_core) + monkeypatch.setattr(smoke, "_request_json", _record_timeout) + + assert smoke.run_smoke( + model_id, 20.0, None, model_sha256=digest, models_dir=models_dir, + wall_clock_budget_s=24 * 3600, # large, so cap() does not clamp + ) == 0 + + ci_ceiling_s = ( + smoke.BUDGETS_MS["full_response_ms"] * 20.0 * smoke.REGRESSION_TOLERANCE + ) / 1000 + assert adapter_timeout["s"] >= ci_ceiling_s * smoke.DEBRIEF_SLOWDOWN_FACTOR + assert post_timeouts and all(t > adapter_timeout["s"] for t in post_timeouts) + + def test_a_deadline_induced_transport_error_is_a_timeout_not_a_crash( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # cap() shrinks every request timeout to the budget that is left, so the + # last request before the deadline dies client-side and _request_json + # reports it as RUNTIME ("the server never answered"). Nothing crashed + # — the budget ran out — and only the TIMEOUT remedy points at + # phase_durations_s, so the verdict has to be TIMEOUT. + import time + + models_dir, model_id, digest = staged_model + base = _fake_core(_debrief()) + + def _overruns(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/api/sessions"): + time.sleep(0.3) # outlast the budget below + raise smoke.SmokeFailure( + smoke.FailureClass.RUNTIME, + f"{url} did not answer with usable JSON after {timeout:.0f} s", + ) + return base(url, payload=payload, timeout=timeout, expect=expect) + + monkeypatch.setattr(smoke, "_request_json", _overruns) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir, + wall_clock_budget_s=0.2, + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.TIMEOUT] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failure_class"] == smoke.FailureClass.TIMEOUT + assert any("Wall-clock budget" in f for f in results["failures"]) + + def test_a_crash_still_outranks_an_exhausted_budget( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # Both are true at once; "convsim-core exited with code 1" is the more + # actionable story, so RUNTIME keeps precedence over TIMEOUT. + import time + + models_dir, model_id, digest = staged_model + fake_servers["core"]._returncode = 1 + base = _fake_core(_debrief()) + + def _overruns(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/api/sessions"): + time.sleep(0.3) + raise smoke.SmokeFailure(smoke.FailureClass.PIPELINE, "connection reset") + return base(url, payload=payload, timeout=timeout, expect=expect) + + monkeypatch.setattr(smoke, "_request_json", _overruns) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir, + wall_clock_budget_s=0.2, + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.RUNTIME] + results = json.loads(report.read_text(encoding="utf-8")) + assert "convsim-core exited with code 1" in results["failures"][0] + def test_an_unexpected_harness_bug_is_reported_not_raised( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: @@ -814,3 +977,9 @@ def _ends_after_one(url: str, *, payload=None, timeout=None, expect=(200, 201)) assert exit_code == 0 results = json.loads(report.read_text(encoding="utf-8")) assert len(results["turns"]) == 2 # opening + the one turn that ended it + # The turn pipeline already moved the session to 'Ended', so POST /end + # would answer 409 and the run would die as a bogus `pipeline` failure. + assert not any(url.endswith("/end") for url in base.calls), ( + "must not POST /end to a session the scenario already ended" + ) + assert results["debrief"] is not None From b874219be777bebefd2334a1a35d368ec9f5501f Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 02:20:57 -0400 Subject: [PATCH 05/58] Keep exit 1 meaning "latency regression" and nothing else MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three ways the harness could report the wrong thing, all found by reading the classification contract back against the code. A cut-off HTTP transfer raises http.client.IncompleteRead, which is not an OSError, so the OSError-only handler in _download_with_progress let a truncated 2.5 GB download escape as a bare traceback: exit 1, which EXIT_CODES reads as `budget` — the one class that means "the product is fine, it is just slow" — and left the partial file on disk for the next run to checksum-fail on. Catch anything the transfer raises and classify it as `download`, which is what it is. The pre-run modes (--print-registry-model, --download-only, --verify-only) had no equivalent of run_smoke's catch-all, so an unparseable registry.yaml also exited 1 with no banner and no step summary. Classify unanticipated errors there the way run_smoke classifies its own, so exit 1 keeps meaning only what the table claims. Per-turn used_fallback flags come from the best-effort debug endpoint. When they were unavailable, every generated turn read as "not a fallback" and the run passed claiming real NPC turns — a canned safe utterance is a non-empty utterance too, so nothing else distinguishes them. Losing a diagnostic should not turn a healthy nightly red, so warn rather than fail, but say in the report that the proof is missing instead of implying it held. Tests for all three, plus the CLI entry point, which had none. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 5 + docs/real-model-smoke.md | 5 + scripts/nightly-model-smoke.py | 48 +++++- tests/scripts/test_nightly_model_smoke.py | 152 ++++++++++++++++++ 4 files changed, 208 insertions(+), 2 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 0ba6f2df..970864f4 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -70,6 +70,11 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. +Anything the harness did not anticipate — a malformed `registry.yaml`, a bug in +the harness itself — is also reported as exit 5, with a message that says so, so +that **exit 1 only ever means a latency regression**. Nothing exits with an +unclassified traceback. + When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 543f97af..e4fee07b 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -66,6 +66,11 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. +Anything the harness did not anticipate — a malformed `registry.yaml`, a bug in +the harness itself — is also reported as exit 5, with a message that says so, so +that **exit 1 only ever means a latency regression**. Nothing exits with an +unclassified traceback. + When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index edb4c72a..0b2ed6f7 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -381,14 +381,20 @@ def _download_with_progress(url: str, dest: Path) -> None: flush=True, ) print() - except OSError as exc: + except Exception as exc: # noqa: BLE001 — see below # Network failure, HTTP error, or a full disk — all "could not fetch the # model", none of which say anything about the app under test. Remove the # partial file so a re-run does not checksum-fail on a truncated download. + # + # Deliberately broader than OSError: a cut-off 2.5 GB response raises + # http.client.IncompleteRead, which is NOT an OSError, so an OSError-only + # handler let a truncated download escape as a bare traceback — exit 1, + # which EXIT_CODES reads as `budget` — and left the partial file on disk + # for the next run to checksum-fail on. dest.unlink(missing_ok=True) raise SmokeFailure( FailureClass.DOWNLOAD, - f"Download failed for {url[:120]}: {exc}", + f"Download failed for {url[:120]}: {exc!r}", ) from exc @@ -586,6 +592,10 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: and the product's job is to recover from that — but if *every* generated turn fell back to the canned safe utterance then no real NPC turns were produced and the smoke proved nothing, so that is a failure. + + Turns carrying no ``used_fallback`` flag at all — the debug endpoint is + best-effort — warn that the check did not run, so an absent proof is not + mistaken for a passed one. """ failures: List[str] = [] warnings: List[str] = [] @@ -602,6 +612,17 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: failures.append("No model-generated NPC turns were recorded") return (failures, warnings) + if not any("used_fallback" in t for t in generated): + # The flags come from the debug endpoint, which is best-effort. Without + # them a canned fallback is indistinguishable from real model output — + # it is a non-empty NPC utterance too — so the check below silently + # passes. Say so, rather than let the report imply a proof we do not have. + warnings.append( + "Per-turn parse flags were unavailable, so the fallback check did not " + "run: this run cannot prove the NPC turns were model output rather " + "than the canned safe utterance" + ) + fallbacks = [t for t in generated if t.get("used_fallback")] if len(fallbacks) == len(generated): failures.append( @@ -1220,6 +1241,29 @@ def main(argv: Optional[Sequence[str]] = None) -> int: }) return exc.exit_code + # The pre-run modes (registry lookup, download, verify) have no run_smoke + # around them to catch what they did not anticipate — an unparseable + # registry.yaml, a missing file, a bad --models-dir. Letting those escape + # exits 1, which EXIT_CODES reads as `budget`: the one class that says "the + # product is fine, it is just slow". Classify them the way run_smoke + # classifies its own surprises instead, so a banner and a step summary are + # still written and exit 1 keeps meaning only what it claims to mean. + except Exception as exc: # noqa: BLE001 — anything unanticipated, reported not raised + failures = [ + f"Unexpected error before the smoke started: {exc!r} — this is most " + "likely a bug in the smoke harness or a malformed " + "model-registry/registry.yaml, not a product failure" + ] + _print_banner(FailureClass.PIPELINE, failures) + _write_step_summary({ + "verdict": "fail", + "model_id": args.model_id, + "failure_class": FailureClass.PIPELINE, + "exit_code": EXIT_CODES[FailureClass.PIPELINE], + "failures": failures, + }) + return EXIT_CODES[FailureClass.PIPELINE] + return run_smoke( args.model_id, args.ci_hardware_factor, diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index c6ac801c..aac1d653 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -12,6 +12,7 @@ from __future__ import annotations import hashlib +import http.client import importlib.util import json from pathlib import Path @@ -133,6 +134,56 @@ def _never(*args: object, **kwargs: object) -> None: assert exc_info.value.failure_class == smoke.FailureClass.CHECKSUM +class TestDownloadFailureClassification: + """Every way a fetch can break has to land in the `download` class.""" + + @staticmethod + def _urlopen_raising(exc: BaseException): + class _Resp: + headers = {"Content-Length": "1048576"} + + def read(self, _n: int) -> bytes: + raise exc + + def __enter__(self): + return self + + def __exit__(self, *a: object) -> bool: + return False + + return lambda *args, **kwargs: _Resp() + + @pytest.mark.parametrize("exc", [ + # A cut-off response mid-body. NOT an OSError, so an OSError-only + # handler let it escape as a traceback — exit 1, which reads as `budget`. + http.client.IncompleteRead(b"partial"), + ConnectionResetError("peer hung up"), + TimeoutError("read timed out"), + ]) + def test_a_broken_transfer_is_a_download_failure( + self, exc: BaseException, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + smoke.urllib.request, "urlopen", self._urlopen_raising(exc) + ) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._download_with_progress("https://example.invalid/m.gguf", tmp_path / "m.gguf") + assert exc_info.value.failure_class == smoke.FailureClass.DOWNLOAD + assert exc_info.value.exit_code == 2 + + def test_a_broken_transfer_leaves_no_partial_file_to_checksum_fail_on( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + dest = tmp_path / "m.gguf" + monkeypatch.setattr( + smoke.urllib.request, "urlopen", + self._urlopen_raising(http.client.IncompleteRead(b"partial")), + ) + with pytest.raises(smoke.SmokeFailure): + smoke._download_with_progress("https://example.invalid/m.gguf", dest) + assert not dest.exists() + + # --------------------------------------------------------------------------- # Registry resolution # --------------------------------------------------------------------------- @@ -250,6 +301,21 @@ def test_authored_opening_does_not_count_as_a_generated_turn(self) -> None: failures, _ = smoke.evaluate_turns(turns) assert any("fell back" in f for f in failures) + def test_missing_parse_flags_warn_that_the_fallback_check_did_not_run(self) -> None: + # The flags come from the best-effort debug endpoint. Without them a + # canned fallback looks exactly like real model output (a non-empty + # utterance), so the run must not imply a proof it does not have. + turns = [{"label": "player_turn_1", "turn_number": 1, + "model_generated": True, "npc_excerpt": "Go on."}] + failures, warnings = smoke.evaluate_turns(turns) + assert failures == [] + assert any("fallback check did not run" in w for w in warnings) + + def test_present_parse_flags_do_not_warn(self) -> None: + failures, warnings = smoke.evaluate_turns([_turn()]) + assert failures == [] + assert warnings == [] + def test_opening_only_run_fails(self) -> None: turns = [{"label": "npc_opening", "turn_number": 0, "model_generated": False, "npc_excerpt": "Thanks for coming in."}] @@ -983,3 +1049,89 @@ def _ends_after_one(url: str, *, payload=None, timeout=None, expect=(200, 201)) "must not POST /end to a session the scenario already ended" ) assert results["debrief"] is not None + + def test_unavailable_debug_flags_warn_instead_of_silently_passing( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # The debug endpoint is a diagnostic, so losing it must not turn a + # healthy run red — but it is also the only thing that tells a real NPC + # turn from the canned fallback, so the report has to say the check + # did not run. + models_dir, model_id, digest = staged_model + base = _fake_core(_debrief()) + + def _no_debug(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/debug"): + raise smoke.SmokeFailure(smoke.FailureClass.PIPELINE, "HTTP 404") + return base(url, payload=payload, timeout=timeout, expect=expect) + + monkeypatch.setattr(smoke, "_request_json", _no_debug) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == 0 + results = json.loads(report.read_text(encoding="utf-8")) + assert any("fallback check did not run" in w for w in results["warnings"]) + + +# --------------------------------------------------------------------------- +# CLI entry point +# --------------------------------------------------------------------------- + + +class TestMainEntryPoint: + """exit 1 means `budget` and nothing else, so nothing may escape unclassified.""" + + def test_registry_lookup_writes_github_output_and_exits_zero( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + out = tmp_path / "gh-output" + monkeypatch.setenv("GITHUB_OUTPUT", str(out)) + assert smoke.main(["--print-registry-model", "starter"]) == 0 + written = dict( + line.split("=", 1) for line in out.read_text(encoding="utf-8").splitlines() + ) + assert len(written["model_sha256"]) == 64 + assert written["model_url"].startswith("https://") + + def test_a_malformed_registry_is_classified_not_a_bare_traceback( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + # yaml raises ParserError, not SmokeFailure. Escaping exits 1, which + # EXIT_CODES reads as `budget` — "the product is fine, just slow" — and + # writes no banner or step summary at all. + registry = tmp_path / "registry.yaml" + registry.write_text("models: [ unterminated\n", encoding="utf-8") + # Point the real resolver at the malformed file (its registry_path + # default is bound at def time, so the module constant cannot be + # patched), so the failure under test is yaml's, not the patch's. + real_resolve = smoke.resolve_registry_model + monkeypatch.setattr( + smoke, "resolve_registry_model", + lambda role, registry_path=None: real_resolve(role, registry), + ) + + summary = tmp_path / "summary.md" + monkeypatch.setenv("GITHUB_STEP_SUMMARY", str(summary)) + + exit_code = smoke.main(["--print-registry-model", "starter"]) + + assert exit_code != smoke.EXIT_CODES[smoke.FailureClass.BUDGET] + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + assert "FAIL" in summary.read_text(encoding="utf-8") + + def test_a_download_failure_keeps_its_own_exit_code_and_summary( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + summary = tmp_path / "summary.md" + monkeypatch.setenv("GITHUB_STEP_SUMMARY", str(summary)) + # --verify-only against an empty directory: no file to hash. + exit_code = smoke.main([ + "--verify-only", "--model-id", "absent", + "--model-sha256", "0" * 64, "--models-dir", str(tmp_path / "empty"), + ]) + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.DOWNLOAD] + assert "`download`" in summary.read_text(encoding="utf-8") From deb0b3148b8072751170b5f24f6704dfcaf0325d Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 02:30:49 -0400 Subject: [PATCH 06/58] Stop convsim-core before deleting the directory it is writing to MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit run_smoke held the throwaway data directory in a `with tempfile.TemporaryDirectory(...)` nested inside the try block, but the children are stopped in the enclosing `finally`. A `with` block exits before the `finally` of the try it sits in, so the ordering was: delete the data directory, *then* terminate convsim-core — the server was still live, holding its SQLite database, WAL and log files open in a tree being rmtree'd. Unlinking open files is harmless on Linux, but any file the server creates during that window makes rmtree raise, and run_smoke's catch-all reports that as an unexpected harness error: exit 5, class `pipeline`, on a run whose conversation and debrief had both already passed. That is precisely the kind of unattributable red nightly this harness exists to prevent. Create the directory with mkdtemp and remove it in the same `finally`, after both children are stopped, with errors ignored — a stray file left behind is a tidiness problem, not a reason to overturn a decided verdict (the same reason Deadline.finish() does not raise). Covered by a new test asserting the directory still exists when convsim-core is terminated and is gone once run_smoke returns. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 375 +++++++++++----------- tests/scripts/test_nightly_model_smoke.py | 38 +++ 2 files changed, 231 insertions(+), 182 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 0b2ed6f7..80e3649e 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -86,6 +86,7 @@ class exit meaning import hashlib import json import os +import shutil import statistics import subprocess import sys @@ -880,6 +881,15 @@ def run_smoke( llama_stderr_tail: deque = deque(maxlen=200) core_stderr_tail: deque = deque(maxlen=200) + # convsim-core's throwaway data directory (SQLite db, WAL, logs). Created + # and removed by hand rather than with a `with TemporaryDirectory(...)` + # inside the try: that block exits *before* the finally below, so the + # directory would be deleted while convsim-core is still writing to it, and + # an rmtree that lost that race would surface as an unexpected-error + # `pipeline` verdict on an otherwise green run. The finally stops both + # children first, then removes the tree with errors ignored — cleanup must + # never overturn a verdict. + data_dir = Path(tempfile.mkdtemp(prefix="convsim-smoke-")) try: # ── checksum ────────────────────────────────────────────────────────── clock.enter("checksum") @@ -897,201 +907,198 @@ def run_smoke( "No --model-sha256 given: checksum drift was not checked before the run" ) - with tempfile.TemporaryDirectory(prefix="convsim-smoke-") as tmp: - data_dir = Path(tmp) - - # ── runtime ─────────────────────────────────────────────────────── - clock.enter("runtime_start") - print(f"\n[smoke] Starting llama-server on port {LLAMA_SERVER_PORT} " - f"with model: {model_path.name}") - llama_proc = _start_llama_server(model_path, LLAMA_SERVER_PORT) - threading.Thread(target=_drain, args=(llama_proc.stderr, llama_stderr_tail), daemon=True).start() - threading.Thread(target=_drain, args=(llama_proc.stdout,), daemon=True).start() - - print("[smoke] Waiting for llama-server (model load)…") - _wait_for_http( - f"http://127.0.0.1:{LLAMA_SERVER_PORT}/v1/models", - timeout_s=clock.cap(300.0), - label="llama-server", - proc=llama_proc, - ) - print("[smoke] llama-server ready.") - - print(f"[smoke] Starting convsim-core on port {CORE_PORT}…") - core_proc = _start_core(data_dir, CORE_PORT, LLAMA_SERVER_PORT, llama_timeout_s) - threading.Thread(target=_drain, args=(core_proc.stderr, core_stderr_tail), daemon=True).start() - threading.Thread(target=_drain, args=(core_proc.stdout,), daemon=True).start() - - base = f"http://127.0.0.1:{CORE_PORT}/api" - _wait_for_http( - f"{base}/health", - timeout_s=clock.cap(120.0), - label="convsim-core", - proc=core_proc, + # ── runtime ─────────────────────────────────────────────────────────── + clock.enter("runtime_start") + print(f"\n[smoke] Starting llama-server on port {LLAMA_SERVER_PORT} " + f"with model: {model_path.name}") + llama_proc = _start_llama_server(model_path, LLAMA_SERVER_PORT) + threading.Thread(target=_drain, args=(llama_proc.stderr, llama_stderr_tail), daemon=True).start() + threading.Thread(target=_drain, args=(llama_proc.stdout,), daemon=True).start() + + print("[smoke] Waiting for llama-server (model load)…") + _wait_for_http( + f"http://127.0.0.1:{LLAMA_SERVER_PORT}/v1/models", + timeout_s=clock.cap(300.0), + label="llama-server", + proc=llama_proc, + ) + print("[smoke] llama-server ready.") + + print(f"[smoke] Starting convsim-core on port {CORE_PORT}…") + core_proc = _start_core(data_dir, CORE_PORT, LLAMA_SERVER_PORT, llama_timeout_s) + threading.Thread(target=_drain, args=(core_proc.stderr, core_stderr_tail), daemon=True).start() + threading.Thread(target=_drain, args=(core_proc.stdout,), daemon=True).start() + + base = f"http://127.0.0.1:{CORE_PORT}/api" + _wait_for_http( + f"{base}/health", + timeout_s=clock.cap(120.0), + label="convsim-core", + proc=core_proc, + ) + + # Sanity-check that the real runtime is wired (not the fake default). + health = _request_json(f"{base}/health", timeout=clock.cap(30.0)) + runtime_id = health.get("llm_runtime", {}).get("runtime_id") + results["runtime_id"] = runtime_id + print(f"[smoke] convsim-core ready (runtime_id={runtime_id}).") + if runtime_id != "llama_cpp": + raise SmokeFailure( + FailureClass.PIPELINE, + f"Expected runtime_id 'llama_cpp' but core reports {runtime_id!r}. " + "Check CONVSIM_RUNTIME_ID wiring — the smoke must not run on a fake runtime.", ) - # Sanity-check that the real runtime is wired (not the fake default). - health = _request_json(f"{base}/health", timeout=clock.cap(30.0)) - runtime_id = health.get("llm_runtime", {}).get("runtime_id") - results["runtime_id"] = runtime_id - print(f"[smoke] convsim-core ready (runtime_id={runtime_id}).") - if runtime_id != "llama_cpp": - raise SmokeFailure( - FailureClass.PIPELINE, - f"Expected runtime_id 'llama_cpp' but core reports {runtime_id!r}. " - "Check CONVSIM_RUNTIME_ID wiring — the smoke must not run on a fake runtime.", - ) + # ── conversation ────────────────────────────────────────────────────── + clock.enter("conversation") + session = _request_json(f"{base}/sessions", payload={ + "scenario_id": SCENARIO_ID, + "difficulty": "standard", + "player_role_name": "Smoke Tester", + "language": "en", + "input_mode": "text-only", + "tts_enabled": False, + "show_state_meters": False, + "save_transcript": False, # no transcript file written during smoke runs + "seed": 42, + }, timeout=clock.cap(30.0)) + session_id = session["session_id"] + results["session_id"] = session_id + print(f"[smoke] Session {session_id} created. Starting session…") + + # /start replays the scenario's authored opening line — no model + # call — so it is measured for the documented session-start metric but + # needs no inference-sized timeout. + t0 = time.monotonic() + start = _request_json( + f"{base}/sessions/{session_id}/start", + payload={}, + timeout=clock.cap(60.0), + ) + session_start_ms = (time.monotonic() - t0) * 1000 + opening = _npc_turn_content(start.get("events", [])) + print(f"[smoke] NPC opening (authored) in {session_start_ms:.0f} ms: " + f"{_excerpt(opening)!r}") + results["measured_ms"]["session_start_ms"] = round(session_start_ms) + results["turns"].append({ + "label": "npc_opening", + "turn_number": 0, + "model_generated": False, + "latency_ms": round(session_start_ms), + "npc_excerpt": _excerpt(opening), + }) - # ── conversation ────────────────────────────────────────────────── - clock.enter("conversation") - session = _request_json(f"{base}/sessions", payload={ - "scenario_id": SCENARIO_ID, - "difficulty": "standard", - "player_role_name": "Smoke Tester", - "language": "en", - "input_mode": "text-only", - "tts_enabled": False, - "show_state_meters": False, - "save_transcript": False, # no transcript file written during smoke runs - "seed": 42, - }, timeout=clock.cap(30.0)) - session_id = session["session_id"] - results["session_id"] = session_id - print(f"[smoke] Session {session_id} created. Starting session…") - - # /start replays the scenario's authored opening line — no model - # call — so it is measured for the documented session-start metric but - # needs no inference-sized timeout. + turn_latencies: List[float] = [] + ended_by_scenario = False + for i, player_text in enumerate(SCRIPTED_PLAYER_TURNS, start=1): + clock.check() + print(f"[smoke] Player turn {i}/{len(SCRIPTED_PLAYER_TURNS)}…") t0 = time.monotonic() - start = _request_json( - f"{base}/sessions/{session_id}/start", - payload={}, - timeout=clock.cap(60.0), + turn = _request_json( + f"{base}/sessions/{session_id}/turn", + payload={"content": player_text}, + timeout=clock.cap(request_timeout_s), ) - session_start_ms = (time.monotonic() - t0) * 1000 - opening = _npc_turn_content(start.get("events", [])) - print(f"[smoke] NPC opening (authored) in {session_start_ms:.0f} ms: " - f"{_excerpt(opening)!r}") - results["measured_ms"]["session_start_ms"] = round(session_start_ms) + latency_ms = (time.monotonic() - t0) * 1000 + turn_latencies.append(latency_ms) + npc_text = _npc_turn_content(turn.get("events", [])) + print(f" {latency_ms:.0f} ms | NPC: {_excerpt(npc_text)!r}") results["turns"].append({ - "label": "npc_opening", - "turn_number": 0, - "model_generated": False, - "latency_ms": round(session_start_ms), - "npc_excerpt": _excerpt(opening), + "label": f"player_turn_{i}", + "turn_number": i, + "model_generated": True, + "latency_ms": round(latency_ms), + "npc_excerpt": _excerpt(npc_text), + "ending_type": turn.get("ending_type"), }) - - turn_latencies: List[float] = [] - ended_by_scenario = False - for i, player_text in enumerate(SCRIPTED_PLAYER_TURNS, start=1): - clock.check() - print(f"[smoke] Player turn {i}/{len(SCRIPTED_PLAYER_TURNS)}…") - t0 = time.monotonic() - turn = _request_json( - f"{base}/sessions/{session_id}/turn", - payload={"content": player_text}, - timeout=clock.cap(request_timeout_s), + if turn.get("ending_type"): + # The turn pipeline already moved the session to 'Ended', so + # POST /end would 409 — see end_session() in the sessions + # router. Record that and skip the explicit end below. + ended_by_scenario = True + print(f"[smoke] Scenario ended early (ending_type={turn['ending_type']}); " + "stopping the scripted turns.") + break + + # The median is the headline full-response figure: a single unlucky + # turn (runner steal, a long NPC answer) must not flap the nightly, + # while a real regression moves every turn and so moves the median. + results["measured_ms"]["full_response_ms"] = round(statistics.median(turn_latencies)) + results["measured_ms"]["full_response_max_ms"] = round(max(turn_latencies)) + + # Per-turn parse flags tell us whether the model's structured output + # was accepted or quietly replaced by the safe fallback. + flags = _debug_flags_by_turn(base, session_id, clock.cap(30.0)) + for turn in results["turns"]: + flag = flags.get(turn["turn_number"]) + if flag: + turn["used_fallback"] = bool(flag.get("used_fallback")) + turn["used_native_structured_output"] = bool( + flag.get("used_native_structured_output") ) - latency_ms = (time.monotonic() - t0) * 1000 - turn_latencies.append(latency_ms) - npc_text = _npc_turn_content(turn.get("events", [])) - print(f" {latency_ms:.0f} ms | NPC: {_excerpt(npc_text)!r}") - results["turns"].append({ - "label": f"player_turn_{i}", - "turn_number": i, - "model_generated": True, - "latency_ms": round(latency_ms), - "npc_excerpt": _excerpt(npc_text), - "ending_type": turn.get("ending_type"), - }) - if turn.get("ending_type"): - # The turn pipeline already moved the session to 'Ended', so - # POST /end would 409 — see end_session() in the sessions - # router. Record that and skip the explicit end below. - ended_by_scenario = True - print(f"[smoke] Scenario ended early (ending_type={turn['ending_type']}); " - "stopping the scripted turns.") - break - - # The median is the headline full-response figure: a single unlucky - # turn (runner steal, a long NPC answer) must not flap the nightly, - # while a real regression moves every turn and so moves the median. - results["measured_ms"]["full_response_ms"] = round(statistics.median(turn_latencies)) - results["measured_ms"]["full_response_max_ms"] = round(max(turn_latencies)) - - # Per-turn parse flags tell us whether the model's structured output - # was accepted or quietly replaced by the safe fallback. - flags = _debug_flags_by_turn(base, session_id, clock.cap(30.0)) - for turn in results["turns"]: - flag = flags.get(turn["turn_number"]) - if flag: - turn["used_fallback"] = bool(flag.get("used_fallback")) - turn["used_native_structured_output"] = bool( - flag.get("used_native_structured_output") - ) - # ── debrief ─────────────────────────────────────────────────────── - clock.enter("debrief") - if ended_by_scenario: - print("[smoke] Session already Ended by the scenario; skipping /end.") - else: - print("[smoke] Ending session…") - _request_json( - f"{base}/sessions/{session_id}/end", payload={}, timeout=clock.cap(30.0) - ) - print("[smoke] Generating debrief (real model call)…") - t0 = time.monotonic() - debrief = _request_json( - f"{base}/sessions/{session_id}/debrief", - payload={}, - timeout=clock.cap(request_timeout_s), + # ── debrief ─────────────────────────────────────────────────────────── + clock.enter("debrief") + if ended_by_scenario: + print("[smoke] Session already Ended by the scenario; skipping /end.") + else: + print("[smoke] Ending session…") + _request_json( + f"{base}/sessions/{session_id}/end", payload={}, timeout=clock.cap(30.0) + ) + print("[smoke] Generating debrief (real model call)…") + t0 = time.monotonic() + debrief = _request_json( + f"{base}/sessions/{session_id}/debrief", + payload={}, + timeout=clock.cap(request_timeout_s), + ) + debrief_ms = (time.monotonic() - t0) * 1000 + results["measured_ms"]["debrief_ms"] = round(debrief_ms) + results["debrief"] = { + "outcome": debrief.get("outcome"), + "total_turns": debrief.get("total_turns"), + "scores": debrief.get("scores"), + "overall_score": debrief.get("overall_score"), + "summary_excerpt": _excerpt(debrief.get("summary") or ""), + "strength_count": len(debrief.get("strengths") or []), + "improvement_count": len(debrief.get("improvements") or []), + "turning_point_count": len(debrief.get("turning_points") or []), + "used_fallback": bool(debrief.get("used_fallback")), + "latency_ms": round(debrief_ms), + } + print(f"[smoke] Debrief in {debrief_ms:.0f} ms | " + f"overall_score={debrief.get('overall_score')} " + f"scores={debrief.get('scores')}") + + # ── assertions ──────────────────────────────────────────────────────── + clock.enter("assertions") + turn_failures, turn_warnings = evaluate_turns(results["turns"]) + debrief_failures, debrief_warnings = evaluate_debrief(debrief) + results["warnings"] += turn_warnings + debrief_warnings + pipeline_failures = turn_failures + debrief_failures + if pipeline_failures: + results["failures"] = pipeline_failures + raise SmokeFailure( + FailureClass.PIPELINE, + f"{len(pipeline_failures)} end-to-end assertion(s) failed", ) - debrief_ms = (time.monotonic() - t0) * 1000 - results["measured_ms"]["debrief_ms"] = round(debrief_ms) - results["debrief"] = { - "outcome": debrief.get("outcome"), - "total_turns": debrief.get("total_turns"), - "scores": debrief.get("scores"), - "overall_score": debrief.get("overall_score"), - "summary_excerpt": _excerpt(debrief.get("summary") or ""), - "strength_count": len(debrief.get("strengths") or []), - "improvement_count": len(debrief.get("improvements") or []), - "turning_point_count": len(debrief.get("turning_points") or []), - "used_fallback": bool(debrief.get("used_fallback")), - "latency_ms": round(debrief_ms), - } - print(f"[smoke] Debrief in {debrief_ms:.0f} ms | " - f"overall_score={debrief.get('overall_score')} " - f"scores={debrief.get('scores')}") - - # ── assertions ──────────────────────────────────────────────────── - clock.enter("assertions") - turn_failures, turn_warnings = evaluate_turns(results["turns"]) - debrief_failures, debrief_warnings = evaluate_debrief(debrief) - results["warnings"] += turn_warnings + debrief_warnings - pipeline_failures = turn_failures + debrief_failures - if pipeline_failures: - results["failures"] = pipeline_failures - raise SmokeFailure( - FailureClass.PIPELINE, - f"{len(pipeline_failures)} end-to-end assertion(s) failed", - ) - # ── budget ──────────────────────────────────────────────────────── - clock.enter("budget") - budget_failures, budget_lines = evaluate_budgets( - results["measured_ms"], ci_hardware_factor + # ── budget ──────────────────────────────────────────────────────────── + clock.enter("budget") + budget_failures, budget_lines = evaluate_budgets( + results["measured_ms"], ci_hardware_factor + ) + for line in budget_lines: + print(f" {line}") + if budget_failures: + results["failures"] = budget_failures + raise SmokeFailure( + FailureClass.BUDGET, f"{len(budget_failures)} latency budget(s) exceeded" ) - for line in budget_lines: - print(f" {line}") - if budget_failures: - results["failures"] = budget_failures - raise SmokeFailure( - FailureClass.BUDGET, f"{len(budget_failures)} latency budget(s) exceeded" - ) - results["verdict"] = "pass" - results["exit_code"] = 0 + results["verdict"] = "pass" + results["exit_code"] = 0 except SmokeFailure as exc: # Precedence, strongest evidence first: @@ -1152,6 +1159,10 @@ def run_smoke( proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() + # Only now that convsim-core is stopped is its data directory safe to + # remove. errors ignored: a stray file left behind is a tidiness + # problem, not a reason to turn a decided verdict into a traceback. + shutil.rmtree(data_dir, ignore_errors=True) results["phase_durations_s"] = clock.finish() results["wall_clock_s"] = round(clock.elapsed_s, 1) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index aac1d653..0109b899 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -739,6 +739,44 @@ def test_both_children_are_stopped_even_on_a_pass( assert fake_servers["llama"].terminated assert fake_servers["core"].terminated + def test_the_data_directory_outlives_convsim_core_and_is_then_removed( + self, staged_model, monkeypatch: pytest.MonkeyPatch + ) -> None: + """The throwaway data dir must not be deleted while core is still running. + + It holds convsim-core's SQLite database, WAL and logs. Removing it + before the child is stopped races the server's own writes, and an + rmtree that lost that race would be caught as an unexpected harness + error and reported as a `pipeline` failure on an otherwise green run. + """ + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core(_debrief())) + seen: dict = {} + + core = _FakeProc() + + def _terminate() -> None: + seen["existed_at_terminate"] = seen["data_dir"].exists() + core.terminated = True + + core.terminate = _terminate # type: ignore[method-assign] + + def _start_core(data_dir, *args, **kwargs) -> _FakeProc: + seen["data_dir"] = Path(data_dir) + return core + + monkeypatch.setattr(smoke, "_start_llama_server", lambda *a, **k: _FakeProc()) + monkeypatch.setattr(smoke, "_start_core", _start_core) + monkeypatch.setattr(smoke, "_wait_for_http", lambda *a, **k: None) + + exit_code = smoke.run_smoke( + model_id, 20.0, None, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == 0 + assert seen["existed_at_terminate"] is True + assert not seen["data_dir"].exists(), "the data directory leaked" + def test_unscored_debrief_is_a_pipeline_failure( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: From ff9ec5f26e2a74b600c07de99ea79c669b391fae Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 02:34:08 -0400 Subject: [PATCH 07/58] Stop a bad command line from reading as a download failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit argparse exits 2 on a usage error, and 2 is the `download` code — whose remedy is "nothing about the app changed, re-run the job". A triager following that advice re-runs a job that cannot succeed, because the problem is the invocation. The module docstring and docs/real-model-smoke.md both promise that every exit code names a class and that nothing escapes unclassified; exit 2 with no banner contradicted both. Route usage errors through the `pipeline` class, like every other surprise the harness did not anticipate. That exposed a second, smaller dishonesty: three failures now borrow `pipeline`'s exit code without sharing its cause (a usage error, a malformed registry.yaml, a harness bug mid-run), and all three printed the class remedy, "inspect the per-turn used_fallback flags" — pointing triage at model output when the thing to read is the traceback or the usage line. _print_banner and the step summary now take a per-failure remedy override, and each of the three supplies the advice that actually applies. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 9 +-- docs/real-model-smoke.md | 9 +-- scripts/nightly-model-smoke.py | 58 ++++++++++++++++--- tests/scripts/test_nightly_model_smoke.py | 15 +++++ 4 files changed, 76 insertions(+), 15 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 970864f4..ab407d27 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -70,10 +70,11 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. -Anything the harness did not anticipate — a malformed `registry.yaml`, a bug in -the harness itself — is also reported as exit 5, with a message that says so, so -that **exit 1 only ever means a latency regression**. Nothing exits with an -unclassified traceback. +Anything the harness did not anticipate — a malformed `registry.yaml`, a bad +command line, a bug in the harness itself — is also reported as exit 5, with a +message and a remedy that say so, so that **exit 1 only ever means a latency +regression**. Nothing exits with an unclassified traceback, and nothing borrows +a class's exit code without correcting the advice printed beneath it. When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index e4fee07b..ef62ea5a 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -66,10 +66,11 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. -Anything the harness did not anticipate — a malformed `registry.yaml`, a bug in -the harness itself — is also reported as exit 5, with a message that says so, so -that **exit 1 only ever means a latency regression**. Nothing exits with an -unclassified traceback. +Anything the harness did not anticipate — a malformed `registry.yaml`, a bad +command line, a bug in the harness itself — is also reported as exit 5, with a +message and a remedy that say so, so that **exit 1 only ever means a latency +regression**. Nothing exits with an unclassified traceback, and nothing borrows +a class's exit code without correcting the advice printed beneath it. When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 80e3649e..c11008e5 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -97,7 +97,7 @@ class exit meaning import urllib.request from collections import deque from pathlib import Path -from typing import Any, Dict, List, Optional, Sequence +from typing import Any, Dict, List, NoReturn, Optional, Sequence REPO_ROOT = Path(__file__).parent.parent REGISTRY_PATH = REPO_ROOT / "model-registry" / "registry.yaml" @@ -725,7 +725,7 @@ def render_step_summary(results: Dict[str, Any]) -> str: *[f"- {f}" for f in results.get("failures", [])], "", "### What to do", - REMEDIES.get(cls, "No remedy recorded."), + results.get("remedy") or REMEDIES.get(cls, "No remedy recorded."), ] measured = results.get("measured_ms") or {} if measured: @@ -757,7 +757,15 @@ def _write_step_summary(results: Dict[str, Any]) -> None: print(f"[smoke] Could not write step summary: {exc}", file=sys.stderr) -def _print_banner(failure_class: str, failures: Sequence[str]) -> None: +def _print_banner( + failure_class: str, failures: Sequence[str], *, remedy: Optional[str] = None +) -> None: + """Print the failure banner. ``remedy`` overrides the class default. + + Some failures borrow a class's exit code without matching its usual cause — + a harness bug reported as ``pipeline``, say — and printing "inspect the + per-turn used_fallback flags" under those sends triage the wrong way. + """ bar = "━" * 72 print(f"\n{bar}", file=sys.stderr) print(f"[smoke] FAILED — class: {failure_class.upper()} " @@ -765,7 +773,7 @@ def _print_banner(failure_class: str, failures: Sequence[str]) -> None: print(bar, file=sys.stderr) for f in failures: print(f" ✗ {f}", file=sys.stderr) - print(f"\n → {REMEDIES[failure_class]}", file=sys.stderr) + print(f"\n → {remedy or REMEDIES[failure_class]}", file=sys.stderr) print(f"{bar}\n", file=sys.stderr) @@ -1147,8 +1155,16 @@ def run_smoke( "a bug in the smoke harness itself, not in the product " "(tests/scripts/test_nightly_model_smoke.py covers its logic)" ] + # Borrowing `pipeline`'s exit code, not its cause: the class remedy + # ("inspect the per-turn used_fallback flags") would point triage at + # model output when the traceback above is the thing to read. + results["remedy"] = ( + "Read the traceback above and the phase it names. This is a harness " + "bug, not a product verdict — add a case to " + "tests/scripts/test_nightly_model_smoke.py once you have it." + ) _dump_stderr_tails((("convsim-core", core_stderr_tail), ("llama-server", llama_stderr_tail))) - _print_banner(FailureClass.PIPELINE, results["failures"]) + _print_banner(FailureClass.PIPELINE, results["failures"], remedy=results["remedy"]) finally: for proc in (core_proc, llama_proc): @@ -1183,8 +1199,30 @@ def run_smoke( # ── CLI ─────────────────────────────────────────────────────────────────────── +class _ClassifiedArgumentParser(argparse.ArgumentParser): + """An ``ArgumentParser`` whose usage errors land in a known failure class. + + argparse exits 2 on a bad command line, and EXIT_CODES reads 2 as + ``download`` — whose remedy is "nothing about the app changed, re-run the + job". That would send a triager round a loop re-running a job that cannot + succeed. A malformed invocation is a harness problem, which is exactly what + ``pipeline`` means for everything else the harness did not anticipate. + """ + + def error(self, message: str) -> NoReturn: + self.print_usage(sys.stderr) + _print_banner( + FailureClass.PIPELINE, + [f"Invalid command line: {message}"], + remedy="Fix the invocation. See the usage above, the examples in this " + "file's docstring, or the smoke step in " + ".github/workflows/model-smoke-nightly.yml.", + ) + sys.exit(EXIT_CODES[FailureClass.PIPELINE]) + + def main(argv: Optional[Sequence[str]] = None) -> int: - parser = argparse.ArgumentParser( + parser = _ClassifiedArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter ) parser.add_argument("--print-registry-model", metavar="ROLE", default=None, @@ -1265,13 +1303,19 @@ def main(argv: Optional[Sequence[str]] = None) -> int: "likely a bug in the smoke harness or a malformed " "model-registry/registry.yaml, not a product failure" ] - _print_banner(FailureClass.PIPELINE, failures) + remedy = ( + "Nothing was played, so this says nothing about the product. Re-run " + "the harness locally with the same arguments (docs/real-model-smoke.md) " + "and fix the registry file or the harness bug in the traceback above." + ) + _print_banner(FailureClass.PIPELINE, failures, remedy=remedy) _write_step_summary({ "verdict": "fail", "model_id": args.model_id, "failure_class": FailureClass.PIPELINE, "exit_code": EXIT_CODES[FailureClass.PIPELINE], "failures": failures, + "remedy": remedy, }) return EXIT_CODES[FailureClass.PIPELINE] diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 0109b899..31b74d50 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -1057,6 +1057,11 @@ def _bug(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: assert exit_code != 0 results = json.loads(report.read_text(encoding="utf-8")) assert "bug in the smoke harness" in results["failures"][0] + # It borrows `pipeline`'s exit code but not its cause, so the remedy + # must point at the traceback, not at the model's turn output. + assert results["remedy"] != smoke.REMEDIES[smoke.FailureClass.PIPELINE] + assert "harness bug" in results["remedy"] + assert results["remedy"] in smoke.render_step_summary(results) def test_a_scenario_that_ends_early_stops_the_scripted_turns( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path @@ -1173,3 +1178,13 @@ def test_a_download_failure_keeps_its_own_exit_code_and_summary( ]) assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.DOWNLOAD] assert "`download`" in summary.read_text(encoding="utf-8") + + def test_a_usage_error_is_not_mistaken_for_a_download_failure(self) -> None: + # argparse exits 2 on a bad command line, and 2 is the `download` code, + # whose remedy is "nothing about the app changed, re-run the job" — + # advice that loops forever on a job that cannot succeed. A malformed + # invocation is a harness problem: `pipeline`, like every other surprise. + with pytest.raises(SystemExit) as exc_info: + smoke.main(["--download-only", "--model-id", "x"]) # no url / sha256 + assert exc_info.value.code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + assert exc_info.value.code != smoke.EXIT_CODES[smoke.FailureClass.DOWNLOAD] From ecdd1b1e5b54b8f90fce2a9ca75d984601a04c1d Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 06:09:53 -0400 Subject: [PATCH 08/58] Blame the phase that ran long, not the one that never started MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deadline.enter() switched phases and then checked the budget, so an overrun noticed at a phase boundary was attributed to the phase about to begin. A conversation that ate the whole budget reported "Wall-clock budget exhausted during phase 'debrief'" before the debrief had executed an instruction, and failed_phase in the report said the same — pointing triage at the wrong code in the one failure class whose entire purpose is naming where the time went. (Overruns noticed mid-phase, via clock.check() and cap(), were always attributed correctly; only the boundary case was wrong.) Check first, while self._phase is still the phase that consumed the budget, then switch. finish() in run_smoke's finally still closes out that phase, so phase_durations_s now also credits the elapsed time to the phase that spent it rather than splitting it across the boundary. Also record the class's remedy in the JSON report. The failure table in docs/real-model-smoke.md sends a triager to the report artifact for several classes; it carried failure_class but not the advice that goes with it, so the mapping had to be looked up elsewhere. The step summary already printed it. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 16 ++++++++++++++-- tests/scripts/test_nightly_model_smoke.py | 20 ++++++++++++++++++-- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index c11008e5..645163be 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -267,9 +267,16 @@ def phase(self) -> str: return self._phase def enter(self, phase: str) -> None: - """Close out the current phase's duration, start a new one, check the budget.""" - self._close_phase(phase) + """Check the budget, then close out this phase's duration and start ``phase``. + + The check comes first, while ``self._phase`` is still the phase that has + been running: it is the one that consumed the budget, and so the one a + triager needs named. Switching phases first would attribute the overrun + to a phase that has not executed a single instruction — reporting + "exhausted during phase 'debrief'" for a conversation that ran long. + """ self.check() + self._close_phase(phase) def finish(self) -> Dict[str, float]: """Close the books without checking the budget. @@ -1130,6 +1137,11 @@ def run_smoke( results["failure_class"] = failure_class results["exit_code"] = EXIT_CODES[failure_class] results["failed_phase"] = exc.phase or clock.phase + # Record the remedy in the artifact too: the failure table in + # docs/real-model-smoke.md sends triage to the JSON report for several + # classes, and a report that names a class without the advice attached + # makes the reader go and look the mapping up again. + results["remedy"] = REMEDIES[failure_class] if not results["failures"]: results["failures"] = [str(exc)] if crashed: diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 31b74d50..3636e639 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -485,11 +485,24 @@ def test_phase_durations_are_recorded(self) -> None: def test_exhausted_budget_raises_timeout_naming_the_phase(self) -> None: clock = smoke.Deadline(0.0) with pytest.raises(smoke.SmokeFailure) as exc_info: - clock.enter("debrief") + clock.check() assert exc_info.value.failure_class == smoke.FailureClass.TIMEOUT assert exc_info.value.exit_code == 6 # The phase recorded is the one that ran out of budget. - assert "debrief" in str(exc_info.value) + assert exc_info.value.phase == "startup" + assert "startup" in str(exc_info.value) + + def test_an_overrun_found_at_a_boundary_blames_the_phase_that_ran_long(self) -> None: + # The remedy for a timeout is "look at phase_durations_s", so the phase + # it names has to be the one that consumed the budget. Blaming the phase + # about to start would point triage at code that never executed. + clock = smoke.Deadline(0.0) + clock._phase = "conversation" # the phase that overran + with pytest.raises(smoke.SmokeFailure) as exc_info: + clock.enter("debrief") + assert exc_info.value.phase == "conversation" + assert "conversation" in str(exc_info.value) + assert "debrief" not in str(exc_info.value) def test_cap_clamps_a_request_timeout_to_the_remaining_budget(self) -> None: clock = smoke.Deadline(30.0) @@ -796,6 +809,9 @@ def test_unscored_debrief_is_a_pipeline_failure( assert results["failure_class"] == smoke.FailureClass.PIPELINE assert results["failed_phase"] == "assertions" assert any("rubric dimension scores" in f for f in results["failures"]) + # The artifact carries the advice, not just the class: the failure table + # in the docs sends triage here for a `pipeline` verdict. + assert results["remedy"] == smoke.REMEDIES[smoke.FailureClass.PIPELINE] def test_latency_regression_is_a_budget_failure_not_a_pipeline_one( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path From 50f9bed0473b69513f514f73c696aefecf7b9c00 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 06:10:09 -0400 Subject: [PATCH 09/58] Count the cache save against the job clock, and stop calling projections measurements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two accuracy problems in the runtime-budget accounting. The pre-smoke tally — the steps that burn timeout-minutes before the harness deadline starts ticking — listed checkout, pip install, cache restore and the cold 2.5 GB download, but not the `actions/cache/save` step this PR introduced by splitting restore from save. Uploading 2.5 GB of incompressible GGUF is ~1 min, so the cold-path figure is ~9 min, not ~8, leaving ~1 min of margin under the 30 min job timeout rather than ~2. The guard test asserted the understated number, so it would have let a future budget increase through. And the per-phase table was presented as "Measured on ubuntu-latest" when only the download, verification, cache and model-load rows are. The two inference rows are projected from the single latency this job has ever measured (~116 s for one behavioral-interview turn); the multi-turn conversation and the debrief have never run on a runner. Marked as projections, with a note to replace them with phase_durations_s from the first green nightly — a budget table nobody can tell the measured rows from is a table nobody will correct. Co-Authored-By: Claude Opus 5 --- .github/workflows/model-smoke-nightly.yml | 20 +++++++++++----- .../src/content/docs/dev/real-model-smoke.md | 23 ++++++++++++------- docs/real-model-smoke.md | 23 ++++++++++++------- scripts/nightly-model-smoke.py | 5 ++-- tests/scripts/test_nightly_model_smoke.py | 6 ++--- 5 files changed, 50 insertions(+), 27 deletions(-) diff --git a/.github/workflows/model-smoke-nightly.yml b/.github/workflows/model-smoke-nightly.yml index f142059b..5d58e30e 100644 --- a/.github/workflows/model-smoke-nightly.yml +++ b/.github/workflows/model-smoke-nightly.yml @@ -35,18 +35,26 @@ # timeout-minutes: 30 is the hard ceiling; the script's own # --wall-clock-budget-s 1200 (20 min) trips first so the failure is attributed # to a phase instead of surfacing as an unattributable job cancellation. The -# 10 min gap is not slack: the steps before the smoke (checkout, pip install, -# cache restore, a cold 2.5 GB download) run against the same job clock. -# Observed breakdown on ubuntu-latest (CPU-only): +# 10 min gap is not slack: every step before the smoke (checkout, pip install, +# cache restore and, on a cache miss, a 2.5 GB download plus the cache save) +# runs against the same job clock and is invisible to the script's own +# deadline — up to ~9 min of it on a cold night. # # pip install (cached wheels) ~2 min # model download (cache miss only) ~2 min — ~0 on a cache hit # SHA-256 verification (2.5 GB, ×2 cold) ~1 min — ~0.5 min on a cache hit +# cache save (cache miss only) ~1 min — ~0 on a cache hit # llama-server model load ~0.5 min -# 3 scripted turns + authored opening ~7 min -# debrief generation ~4 min +# 3 scripted turns + authored opening ~7 min — projected, see below +# debrief generation ~4 min — projected, see below # ------------------------------------------------ -# total ~16 min warm cache, ~18 min cold +# total ~16 min warm cache, ~19 min cold +# +# The download, verification, cache and model-load figures are measured; the +# inference figures are projected from the one latency this job has actually +# measured (~116 s per behavioral-interview turn, see below) and should be +# replaced with real per-phase durations from the first green nightly's report +# artifact. # # If the smoke step creeps past ~15 min — or the job past ~25 min — reduce # SCRIPTED_PLAYER_TURNS rather than raising either ceiling: a nightly that diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index ab407d27..a1d4f96d 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -102,22 +102,29 @@ canceled". The 10-minute gap between the two is not slack. `timeout-minutes` covers the whole job, and the harness only starts after checkout, three `pip install` -steps, the cache restore and — on a cache miss — a 2.5 GB download: up to -~8 min that the harness's own clock never sees. A 25 min harness budget would -lose the race to the job timeout on exactly the cold-cache nights where an -attributed verdict matters most. +steps, the cache restore and — on a cache miss — a 2.5 GB download *and* the +cache save that follows it: up to ~9 min that the harness's own clock never +sees. A 25 min harness budget would lose the race to the job timeout on exactly +the cold-cache nights where an attributed verdict matters most. -Measured on `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): +On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | |---|---|---| | `pip install` (prompt-composer, convsim-core, llama-cpp-python wheel) | ~2 min | ~2 min | | Model download (2.5 GB from Hugging Face) | ~2 min | — | | SHA-256 verification | ~1 min (×2) | ~0.5 min | +| Cache save (2.5 GB) | ~1 min | — | | `llama-server` model load | ~0.5 min | ~0.5 min | -| Authored opening + 3 scripted turns | ~7 min | ~7 min | -| Debrief generation | ~4 min | ~4 min | -| **Total** | **~18 min** | **~16 min** | +| Authored opening + 3 scripted turns † | ~7 min | ~7 min | +| Debrief generation † | ~4 min | ~4 min | +| **Total** | **~19 min** | **~16 min** | + +† The download, verification, cache and model-load rows are measured. The two +inference rows are *projected* from the only latency this job has measured so +far — ~116 s for a single behavioral-interview turn (see below) — because the +multi-turn conversation and the debrief have never run on a runner. Replace them +with the real `phase_durations_s` from the first green nightly's report artifact. The model is cached between runs under the key `model-gguf-v1-`, so the download only recurs when the registry diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index ef62ea5a..0ab3adf4 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -98,22 +98,29 @@ canceled". The 10-minute gap between the two is not slack. `timeout-minutes` covers the whole job, and the harness only starts after checkout, three `pip install` -steps, the cache restore and — on a cache miss — a 2.5 GB download: up to -~8 min that the harness's own clock never sees. A 25 min harness budget would -lose the race to the job timeout on exactly the cold-cache nights where an -attributed verdict matters most. +steps, the cache restore and — on a cache miss — a 2.5 GB download *and* the +cache save that follows it: up to ~9 min that the harness's own clock never +sees. A 25 min harness budget would lose the race to the job timeout on exactly +the cold-cache nights where an attributed verdict matters most. -Measured on `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): +On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | |---|---|---| | `pip install` (prompt-composer, convsim-core, llama-cpp-python wheel) | ~2 min | ~2 min | | Model download (2.5 GB from Hugging Face) | ~2 min | — | | SHA-256 verification | ~1 min (×2) | ~0.5 min | +| Cache save (2.5 GB) | ~1 min | — | | `llama-server` model load | ~0.5 min | ~0.5 min | -| Authored opening + 3 scripted turns | ~7 min | ~7 min | -| Debrief generation | ~4 min | ~4 min | -| **Total** | **~18 min** | **~16 min** | +| Authored opening + 3 scripted turns † | ~7 min | ~7 min | +| Debrief generation † | ~4 min | ~4 min | +| **Total** | **~19 min** | **~16 min** | + +† The download, verification, cache and model-load rows are measured. The two +inference rows are *projected* from the only latency this job has measured so +far — ~116 s for a single behavioral-interview turn (see below) — because the +multi-turn conversation and the debrief have never run on a runner. Replace them +with the real `phase_durations_s` from the first green nightly's report artifact. The model is cached between runs under the key `model-gguf-v1-`, so the download only recurs when the registry diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 645163be..9045ee1f 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -131,8 +131,9 @@ class exit meaning # GitHub killing the job with an unattributable "The operation was canceled", # so this has to clear the job's timeout-minutes *minus the steps that run # before the script does*: checkout, pip install, cache restore and (on a cache -# miss) the 2.5 GB download, together up to ~8 min of the 30 min job. 20 min -# therefore leaves the script's own deadline the first one to trip. +# miss) the 2.5 GB download plus the cache save that follows it, together up to +# ~9 min of the 30 min job. 20 min therefore leaves the script's own deadline +# the first one to trip. # See docs/real-model-smoke.md. DEFAULT_WALL_CLOCK_BUDGET_S = 1200.0 # 20 min diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 3636e639..72f5093b 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -426,9 +426,9 @@ def test_failure_message_shows_the_arithmetic(self) -> None: # Steps that burn the job's clock before the harness starts and so are invisible # to its own deadline: checkout, three pip installs, the cache restore and, on a -# cache miss, the 2.5 GB download. Measured at ~8 min cold; see the breakdown in -# docs/real-model-smoke.md. -PRE_SMOKE_JOB_MINUTES = 8 +# cache miss, the 2.5 GB download plus the cache save that follows it. ~9 min +# cold; see the breakdown in docs/real-model-smoke.md. +PRE_SMOKE_JOB_MINUTES = 9 _WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "model-smoke-nightly.yml" From 42cd2db7512cecc1e44998e0e468a44cd5879774 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 06:22:30 -0400 Subject: [PATCH 10/58] Stop a full disk from reading as a latency regression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit run_smoke() classifies everything inside its own try/finally, but it was called from outside main()'s classifying handler — and its own prologue (the throwaway convsim-core data directory) and epilogue (the report and step-summary writes that follow the verdict) sit outside that block. An OSError there escaped as a bare traceback, so Python exited 1, which EXIT_CODES reads as `budget`: the one class whose remedy says the product is fine and just slow, and sends the triager looking for a prompt-size or sampling regression that does not exist. A full disk is the realistic cause, and the nightly reaches that step right after writing a 2.5 GB download and saving a 2.5 GB cache entry. Move the run_smoke() call inside the handler that already classifies the pre-run modes' surprises, so a banner and a step summary are still written and exit 1 keeps meaning only what it claims to mean. Its message and remedy now cover an escape from either side of run_smoke rather than only "before the smoke started", and the docs name a mid-run disk fill among the unanticipated failures that land on exit 5. Two tests cover it: mkdtemp failing in the prologue, and the report write failing after a verdict was already decided. Both exited 1 before. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 9 ++-- docs/real-model-smoke.md | 9 ++-- scripts/nightly-model-smoke.py | 53 +++++++++++-------- tests/scripts/test_nightly_model_smoke.py | 47 ++++++++++++++++ 4 files changed, 88 insertions(+), 30 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index a1d4f96d..0415a5a4 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -71,10 +71,11 @@ checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. Anything the harness did not anticipate — a malformed `registry.yaml`, a bad -command line, a bug in the harness itself — is also reported as exit 5, with a -message and a remedy that say so, so that **exit 1 only ever means a latency -regression**. Nothing exits with an unclassified traceback, and nothing borrows -a class's exit code without correcting the advice printed beneath it. +command line, a disk that fills up mid-run, a bug in the harness itself — is +also reported as exit 5, with a message and a remedy that say so, so that +**exit 1 only ever means a latency regression**. Nothing exits with an +unclassified traceback, and nothing borrows a class's exit code without +correcting the advice printed beneath it. When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 0ab3adf4..fd25c0b1 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -67,10 +67,11 @@ checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. Anything the harness did not anticipate — a malformed `registry.yaml`, a bad -command line, a bug in the harness itself — is also reported as exit 5, with a -message and a remedy that say so, so that **exit 1 only ever means a latency -regression**. Nothing exits with an unclassified traceback, and nothing borrows -a class's exit code without correcting the advice printed beneath it. +command line, a disk that fills up mid-run, a bug in the harness itself — is +also reported as exit 5, with a message and a remedy that say so, so that +**exit 1 only ever means a latency regression**. Nothing exits with an +unclassified traceback, and nothing borrows a class's exit code without +correcting the advice printed beneath it. When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 9045ee1f..e3904afd 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1292,6 +1292,21 @@ def main(argv: Optional[Sequence[str]] = None) -> int: verify_model_checksum(models_dir / f"{args.model_id}.gguf", args.model_sha256) return 0 + # Inside the handler, not after it: run_smoke classifies everything in + # its own try/finally, but its prologue (the throwaway data directory) + # and its epilogue (the report and step-summary writes that follow the + # verdict) sit outside that block. An OSError there — a full disk is + # the realistic one, right after a 2.5 GB download and a 2.5 GB cache + # save — would escape as a bare traceback and exit 1. + return run_smoke( + args.model_id, + args.ci_hardware_factor, + Path(args.report_path) if args.report_path else None, + model_sha256=args.model_sha256, + models_dir=models_dir, + wall_clock_budget_s=args.wall_clock_budget_s, + ) + except SmokeFailure as exc: _print_banner(exc.failure_class, [str(exc)]) _write_step_summary({ @@ -1303,23 +1318,26 @@ def main(argv: Optional[Sequence[str]] = None) -> int: }) return exc.exit_code - # The pre-run modes (registry lookup, download, verify) have no run_smoke - # around them to catch what they did not anticipate — an unparseable - # registry.yaml, a missing file, a bad --models-dir. Letting those escape - # exits 1, which EXIT_CODES reads as `budget`: the one class that says "the - # product is fine, it is just slow". Classify them the way run_smoke - # classifies its own surprises instead, so a banner and a step summary are - # still written and exit 1 keeps meaning only what it claims to mean. + # Nothing may leave main() unclassified. The pre-run modes (registry + # lookup, download, verify) have no run_smoke around them to catch what + # they did not anticipate — an unparseable registry.yaml, a missing file, a + # bad --models-dir — and run_smoke itself has a prologue and an epilogue + # outside its own handlers. Letting any of that escape exits 1, which + # EXIT_CODES reads as `budget`: the one class that says "the product is + # fine, it is just slow". Classify them the way run_smoke classifies its + # own surprises instead, so a banner and a step summary are still written + # and exit 1 keeps meaning only what it claims to mean. except Exception as exc: # noqa: BLE001 — anything unanticipated, reported not raised failures = [ - f"Unexpected error before the smoke started: {exc!r} — this is most " - "likely a bug in the smoke harness or a malformed " - "model-registry/registry.yaml, not a product failure" + f"Unexpected error outside the smoke's own classification: {exc!r} — " + "this is most likely a bug in the smoke harness, a malformed " + "model-registry/registry.yaml or an unusable disk, not a product failure" ] remedy = ( - "Nothing was played, so this says nothing about the product. Re-run " - "the harness locally with the same arguments (docs/real-model-smoke.md) " - "and fix the registry file or the harness bug in the traceback above." + "This escaped run_smoke's own error handling, so it says nothing about " + "the product. Re-run the harness locally with the same arguments " + "(docs/real-model-smoke.md) and fix the registry file, the environment, " + "or the harness bug in the traceback above." ) _print_banner(FailureClass.PIPELINE, failures, remedy=remedy) _write_step_summary({ @@ -1332,15 +1350,6 @@ def main(argv: Optional[Sequence[str]] = None) -> int: }) return EXIT_CODES[FailureClass.PIPELINE] - return run_smoke( - args.model_id, - args.ci_hardware_factor, - Path(args.report_path) if args.report_path else None, - model_sha256=args.model_sha256, - models_dir=models_dir, - wall_clock_budget_s=args.wall_clock_budget_s, - ) - if __name__ == "__main__": sys.exit(main()) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 72f5093b..5ddf05c1 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -1195,6 +1195,53 @@ def test_a_download_failure_keeps_its_own_exit_code_and_summary( assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.DOWNLOAD] assert "`download`" in summary.read_text(encoding="utf-8") + def test_a_failure_in_run_smokes_prologue_is_classified_not_exit_one( + self, staged_model, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # run_smoke classifies everything inside its own try/finally, but the + # throwaway data directory is created before it and the report is + # written after it. A full disk there — realistic right after a 2.5 GB + # download and a 2.5 GB cache save — must not escape as a bare + # traceback: Python would exit 1, which EXIT_CODES reads as `budget`. + models_dir, model_id, digest = staged_model + + def _no_space(*args: object, **kwargs: object) -> str: + raise OSError(28, "No space left on device") + + monkeypatch.setattr(smoke.tempfile, "mkdtemp", _no_space) + summary = tmp_path / "summary.md" + monkeypatch.setenv("GITHUB_STEP_SUMMARY", str(summary)) + + exit_code = smoke.main([ + "--model-id", model_id, "--model-sha256", digest, + "--models-dir", str(models_dir), + ]) + + assert exit_code != smoke.EXIT_CODES[smoke.FailureClass.BUDGET] + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + text = summary.read_text(encoding="utf-8") + assert "FAIL" in text + assert "No space left on device" in text + + def test_a_failure_writing_the_report_does_not_become_a_budget_verdict( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # The report write follows the verdict, outside run_smoke's handlers. + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core(_debrief())) + + def _no_space(*args: object, **kwargs: object) -> None: + raise OSError(28, "No space left on device") + + monkeypatch.setattr(smoke, "_write_report", _no_space) + + exit_code = smoke.main([ + "--model-id", model_id, "--model-sha256", digest, + "--models-dir", str(models_dir), "--report-path", str(tmp_path / "r.json"), + ]) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + def test_a_usage_error_is_not_mistaken_for_a_download_failure(self) -> None: # argparse exits 2 on a bad command line, and 2 is the `download` code, # whose remedy is "nothing about the app changed, re-run the job" — From dce9529044640e0801090ece41c5114bb9ff05aa Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 06:35:20 -0400 Subject: [PATCH 11/58] Stop a broken registry entry from reading as a flaky download MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `resolve_registry_model` classified a registry that names no `starter` model — or one missing `download.url`/`download.sha256` — as `download`, exit 2. That class's remedy ends "Nothing about the app changed — re-run the job", which is advice that loops forever here: no number of re-runs makes a missing registry entry appear, and nothing was ever fetched for the network to be blamed for. It is the same trap the argparse exit-2 fix closed, reached by a different route. These failures now take `pipeline`, the harness's catch-all for work it could not carry out, and carry their own remedy — fix the registry file and merge — rather than `pipeline`'s default "inspect the per-turn used_fallback flags", which describes a conversation that never ran. docs/real-model-smoke.md already promised exactly this ("a malformed registry.yaml ... is also reported as exit 5"); now the code agrees. - `SmokeFailure` takes an optional per-instance `remedy` override. - `run_smoke` and `main` print and record that remedy instead of the class default. `run_smoke` falls back to the class remedy when a crashed child or an exhausted clock has overruled the class, since the original failure's advice then describes a different failure. - Tests: the override, the new classification for a missing role, an ambiguous role, a missing checksum, and the step summary a registry failure actually writes. Also make the runtime-budget breakdown add up: the rows summed to ~2 min less than the stated totals because checkout and setup-python were never listed. Adds that row and corrects the warm-cache total to ~15 min, in the workflow header and both copies of the doc. Co-Authored-By: Claude Opus 5 --- .github/workflows/model-smoke-nightly.yml | 3 +- .../src/content/docs/dev/real-model-smoke.md | 17 +++-- docs/real-model-smoke.md | 17 +++-- scripts/nightly-model-smoke.py | 63 ++++++++++++++--- tests/scripts/test_nightly_model_smoke.py | 68 ++++++++++++++++++- 5 files changed, 140 insertions(+), 28 deletions(-) diff --git a/.github/workflows/model-smoke-nightly.yml b/.github/workflows/model-smoke-nightly.yml index 5d58e30e..62b568e2 100644 --- a/.github/workflows/model-smoke-nightly.yml +++ b/.github/workflows/model-smoke-nightly.yml @@ -40,6 +40,7 @@ # runs against the same job clock and is invisible to the script's own # deadline — up to ~9 min of it on a cold night. # +# checkout + setup-python ~1 min # pip install (cached wheels) ~2 min # model download (cache miss only) ~2 min — ~0 on a cache hit # SHA-256 verification (2.5 GB, ×2 cold) ~1 min — ~0.5 min on a cache hit @@ -48,7 +49,7 @@ # 3 scripted turns + authored opening ~7 min — projected, see below # debrief generation ~4 min — projected, see below # ------------------------------------------------ -# total ~16 min warm cache, ~19 min cold +# total ~15 min warm cache, ~19 min cold # # The download, verification, cache and model-load figures are measured; the # inference figures are projected from the one latency this job has actually diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 0415a5a4..ecb0ab0a 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -70,12 +70,14 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. -Anything the harness did not anticipate — a malformed `registry.yaml`, a bad -command line, a disk that fills up mid-run, a bug in the harness itself — is -also reported as exit 5, with a message and a remedy that say so, so that -**exit 1 only ever means a latency regression**. Nothing exits with an -unclassified traceback, and nothing borrows a class's exit code without -correcting the advice printed beneath it. +Anything the harness could not carry out — a `registry.yaml` that is malformed +or names no usable `starter` model, a bad command line, a disk that fills up +mid-run, a bug in the harness itself — is also reported as exit 5, with a +message and a remedy that say so, so that **exit 1 only ever means a latency +regression**. Nothing exits with an unclassified traceback, and nothing borrows +a class's exit code without correcting the advice printed beneath it: a broken +registry entry is *not* exit 2, because `download`'s remedy ends "re-run the +job" and no re-run will repair a file in the repository. When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: @@ -112,6 +114,7 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | |---|---|---| +| Checkout + `setup-python` | ~1 min | ~1 min | | `pip install` (prompt-composer, convsim-core, llama-cpp-python wheel) | ~2 min | ~2 min | | Model download (2.5 GB from Hugging Face) | ~2 min | — | | SHA-256 verification | ~1 min (×2) | ~0.5 min | @@ -119,7 +122,7 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | `llama-server` model load | ~0.5 min | ~0.5 min | | Authored opening + 3 scripted turns † | ~7 min | ~7 min | | Debrief generation † | ~4 min | ~4 min | -| **Total** | **~19 min** | **~16 min** | +| **Total** | **~19 min** | **~15 min** | † The download, verification, cache and model-load rows are measured. The two inference rows are *projected* from the only latency this job has measured so diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index fd25c0b1..c5704be0 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -66,12 +66,14 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. -Anything the harness did not anticipate — a malformed `registry.yaml`, a bad -command line, a disk that fills up mid-run, a bug in the harness itself — is -also reported as exit 5, with a message and a remedy that say so, so that -**exit 1 only ever means a latency regression**. Nothing exits with an -unclassified traceback, and nothing borrows a class's exit code without -correcting the advice printed beneath it. +Anything the harness could not carry out — a `registry.yaml` that is malformed +or names no usable `starter` model, a bad command line, a disk that fills up +mid-run, a bug in the harness itself — is also reported as exit 5, with a +message and a remedy that say so, so that **exit 1 only ever means a latency +regression**. Nothing exits with an unclassified traceback, and nothing borrows +a class's exit code without correcting the advice printed beneath it: a broken +registry entry is *not* exit 2, because `download`'s remedy ends "re-run the +job" and no re-run will repair a file in the repository. When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: @@ -108,6 +110,7 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | |---|---|---| +| Checkout + `setup-python` | ~1 min | ~1 min | | `pip install` (prompt-composer, convsim-core, llama-cpp-python wheel) | ~2 min | ~2 min | | Model download (2.5 GB from Hugging Face) | ~2 min | — | | SHA-256 verification | ~1 min (×2) | ~0.5 min | @@ -115,7 +118,7 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | `llama-server` model load | ~0.5 min | ~0.5 min | | Authored opening + 3 scripted turns † | ~7 min | ~7 min | | Debrief generation † | ~4 min | ~4 min | -| **Total** | **~19 min** | **~16 min** | +| **Total** | **~19 min** | **~15 min** | † The download, verification, cache and model-load rows are measured. The two inference rows are *projected* from the only latency this job has measured so diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index e3904afd..1543245e 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -49,6 +49,12 @@ class exit meaning ``timeout`` 6 wall-clock budget exhausted (phase named in the message) =========== ==== ======================================================= +Exit 5 is also the catch-all for everything the harness could not carry out — +a bad command line, an unusable ``registry.yaml``, a full disk, a bug in this +file — so that exit 1 never means anything but a latency regression. Those +failures carry their own remedy instead of ``pipeline``'s, because "inspect the +per-turn used_fallback flags" is the wrong first move for any of them. + Transcript content is fully scripted (see ``SCRIPTED_PLAYER_TURNS``) and holds no user data, so short NPC excerpts are logged and reported — they are the evidence that a real model, not a canned fallback, drove the conversation. @@ -226,12 +232,26 @@ class FailureClass: class SmokeFailure(Exception): - """A failure with a known class, so CI can distinguish *why* the smoke failed.""" + """A failure with a known class, so CI can distinguish *why* the smoke failed. + + ``remedy`` overrides the class default for the failures that borrow a + class's exit code without matching its usual cause. Printing the class + remedy under those sends triage the wrong way — the whole point of the + classification is that the advice next to the banner is actionable. + """ - def __init__(self, failure_class: str, message: str, *, phase: Optional[str] = None) -> None: + def __init__( + self, + failure_class: str, + message: str, + *, + phase: Optional[str] = None, + remedy: Optional[str] = None, + ) -> None: super().__init__(message) self.failure_class = failure_class self.phase = phase + self._remedy = remedy @property def exit_code(self) -> int: @@ -239,7 +259,7 @@ def exit_code(self) -> int: @property def remedy(self) -> str: - return REMEDIES[self.failure_class] + return self._remedy or REMEDIES[self.failure_class] # ── Wall-clock budget ───────────────────────────────────────────────────────── @@ -315,6 +335,20 @@ def cap(self, timeout_s: float) -> float: # ── Registry ────────────────────────────────────────────────────────────────── +# A registry that does not describe a usable model is a repository problem, not +# a network one. It borrows `pipeline`'s exit code — the class this harness +# uses for everything it could not carry out — rather than `download`, whose +# remedy ends "Nothing about the app changed — re-run the job": no number of +# re-runs makes a missing `role: starter` entry appear, and the checksum the +# cache key is built from never existed to drift. +REGISTRY_REMEDY = ( + "model-registry/registry.yaml does not describe a usable model for the " + "requested role, so the job never got as far as fetching anything. " + "Re-running cannot fix it: repair the registry entry (validate with " + "scripts/validate-registry.py) and merge." +) + + def resolve_registry_model(role: str, registry_path: Path = REGISTRY_PATH) -> Dict[str, Any]: """Return ``{id, url, sha256, size_gb}`` for the registry model with ``role``. @@ -328,13 +362,15 @@ def resolve_registry_model(role: str, registry_path: Path = REGISTRY_PATH) -> Di matches = [m for m in registry.get("models", []) if m.get("role") == role] if not matches: raise SmokeFailure( - FailureClass.DOWNLOAD, + FailureClass.PIPELINE, f"No model with role {role!r} in {registry_path}", + remedy=REGISTRY_REMEDY, ) if len(matches) > 1: raise SmokeFailure( - FailureClass.DOWNLOAD, + FailureClass.PIPELINE, f"Registry has {len(matches)} models with role {role!r}; expected exactly one", + remedy=REGISTRY_REMEDY, ) model = matches[0] download = model.get("download", {}) @@ -342,8 +378,9 @@ def resolve_registry_model(role: str, registry_path: Path = REGISTRY_PATH) -> Di url = download.get("url") if not sha256 or not url: raise SmokeFailure( - FailureClass.DOWNLOAD, + FailureClass.PIPELINE, f"Registry model {model.get('id')!r} is missing download.url or download.sha256", + remedy=REGISTRY_REMEDY, ) return { "id": model["id"], @@ -1141,8 +1178,13 @@ def run_smoke( # Record the remedy in the artifact too: the failure table in # docs/real-model-smoke.md sends triage to the JSON report for several # classes, and a report that names a class without the advice attached - # makes the reader go and look the mapping up again. - results["remedy"] = REMEDIES[failure_class] + # makes the reader go and look the mapping up again. The failure's own + # remedy only survives while its class does — once a crashed child or an + # exhausted clock has overruled the class, its advice describes a + # different failure than the one being reported. + results["remedy"] = ( + exc.remedy if failure_class == exc.failure_class else REMEDIES[failure_class] + ) if not results["failures"]: results["failures"] = [str(exc)] if crashed: @@ -1154,7 +1196,7 @@ def run_smoke( f"{results['failed_phase']!r} (elapsed {clock.elapsed_s:.0f} s)", ) _dump_stderr_tails((("convsim-core", core_stderr_tail), ("llama-server", llama_stderr_tail))) - _print_banner(failure_class, results["failures"]) + _print_banner(failure_class, results["failures"], remedy=results["remedy"]) # Exception, not BaseException: a Ctrl-C should still stop the children (the # finally block below) but must not be dressed up as a product failure. @@ -1308,13 +1350,14 @@ def main(argv: Optional[Sequence[str]] = None) -> int: ) except SmokeFailure as exc: - _print_banner(exc.failure_class, [str(exc)]) + _print_banner(exc.failure_class, [str(exc)], remedy=exc.remedy) _write_step_summary({ "verdict": "fail", "model_id": args.model_id, "failure_class": exc.failure_class, "exit_code": exc.exit_code, "failures": [str(exc)], + "remedy": exc.remedy, }) return exc.exit_code diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 5ddf05c1..02f05e6d 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -67,7 +67,16 @@ def test_failure_carries_its_exit_code_and_remedy(self) -> None: exc = smoke.SmokeFailure(smoke.FailureClass.RUNTIME, "boom", phase="runtime_start") assert exc.exit_code == smoke.EXIT_CODES[smoke.FailureClass.RUNTIME] assert exc.phase == "runtime_start" - assert exc.remedy + assert exc.remedy == smoke.REMEDIES[smoke.FailureClass.RUNTIME] + + def test_a_failure_can_override_its_class_remedy(self) -> None: + # Failures that borrow a class's exit code without matching its usual + # cause must not print that class's advice. + exc = smoke.SmokeFailure( + smoke.FailureClass.PIPELINE, "boom", remedy="Do this instead." + ) + assert exc.exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + assert exc.remedy == "Do this instead." # --------------------------------------------------------------------------- @@ -215,12 +224,40 @@ def test_starter_is_the_smallest_registry_model(self) -> None: starter = smoke.resolve_registry_model("starter") assert sizes[starter["id"]] == min(sizes.values()) - def test_unknown_role_is_a_download_failure(self, tmp_path: Path) -> None: + def test_unknown_role_is_not_a_download_failure(self, tmp_path: Path) -> None: + # `download`'s remedy ends "Nothing about the app changed — re-run the + # job", which would send a triager round a loop: no number of re-runs + # makes a missing `role: starter` entry appear. A registry that does not + # describe a usable model is a repository problem, so it takes the + # harness's catch-all class and its own remedy. registry = tmp_path / "registry.yaml" registry.write_text("models: []\n", encoding="utf-8") with pytest.raises(smoke.SmokeFailure) as exc_info: smoke.resolve_registry_model("starter", registry) - assert exc_info.value.failure_class == smoke.FailureClass.DOWNLOAD + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + assert exc_info.value.exit_code == 5 + assert "registry.yaml" in exc_info.value.remedy + assert "re-run the job" not in exc_info.value.remedy.lower() + + def test_two_models_claiming_the_same_role_are_rejected(self, tmp_path: Path) -> None: + # The cache key, the download URL and the verified checksum all come + # from this one lookup, so an ambiguous role must stop the run rather + # than let it pick arbitrarily. + registry = tmp_path / "registry.yaml" + registry.write_text( + "models:\n" + " - id: a\n" + " role: starter\n" + " download: {url: https://example.invalid/a.gguf, sha256: aa}\n" + " - id: b\n" + " role: starter\n" + " download: {url: https://example.invalid/b.gguf, sha256: bb}\n", + encoding="utf-8", + ) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.resolve_registry_model("starter", registry) + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + assert "expected exactly one" in str(exc_info.value) def test_model_without_checksum_is_rejected(self, tmp_path: Path) -> None: registry = tmp_path / "registry.yaml" @@ -235,6 +272,7 @@ def test_model_without_checksum_is_rejected(self, tmp_path: Path) -> None: with pytest.raises(smoke.SmokeFailure) as exc_info: smoke.resolve_registry_model("starter", registry) assert "sha256" in str(exc_info.value) + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE def test_github_output_is_appended(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: out = tmp_path / "gh-output" @@ -1182,6 +1220,30 @@ def test_a_malformed_registry_is_classified_not_a_bare_traceback( assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] assert "FAIL" in summary.read_text(encoding="utf-8") + def test_a_registry_missing_the_role_prints_registry_advice( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + # The step summary is the whole triage surface for a red nightly, so + # the advice in it has to match the failure: `pipeline`'s own remedy + # ("inspect the per-turn used_fallback flags") describes a run that + # never happened here. + registry = tmp_path / "registry.yaml" + registry.write_text("models: []\n", encoding="utf-8") + real_resolve = smoke.resolve_registry_model + monkeypatch.setattr( + smoke, "resolve_registry_model", + lambda role, registry_path=None: real_resolve(role, registry), + ) + summary = tmp_path / "summary.md" + monkeypatch.setenv("GITHUB_STEP_SUMMARY", str(summary)) + + exit_code = smoke.main(["--print-registry-model", "starter"]) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + text = summary.read_text(encoding="utf-8") + assert "registry.yaml" in text + assert "used_fallback" not in text + def test_a_download_failure_keeps_its_own_exit_code_and_summary( self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: From ad612aac8f22f7e1d396a12ad2c6b4a4bd3087fd Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 07:04:30 -0400 Subject: [PATCH 12/58] Test the HTTP layer that decides "crashed server" vs "rejected request" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Issue #457 asks the nightly's failure output to distinguish a runtime crash from a pipeline assertion. Two functions make that call — `_request_json` (5xx ⇒ `runtime`, 4xx ⇒ `pipeline`, transport error ⇒ `runtime`) and `_wait_for_http` (an exited child is named immediately instead of polled for) — and every orchestration test monkeypatched both of them away, so neither ran under test. A mutation that collapses the 5xx/4xx split, or that drops the crashed-child short-circuit, passed the whole suite. Cover both directly. The readiness tests use timeouts far below the 300 s the real model-load wait allows: with the short-circuit removed, the crashed-child case would otherwise stall the per-PR suite for five minutes instead of going red. Verified both mutations now fail in under a second. Also report the last connection error with `!r`, as everything else in the harness does. `str()` on an exception raised with no arguments is the empty string, so a readiness failure could render as "llama-server did not become ready within 300 s ()" — leaving the `runtime` verdict's one diagnostic blank. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 7 +- tests/scripts/test_nightly_model_smoke.py | 219 ++++++++++++++++++++++ 2 files changed, 225 insertions(+), 1 deletion(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 1543245e..9dc38c4a 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -565,7 +565,12 @@ def _wait_for_http(url: str, timeout_s: float, label: str, proc: Optional[subpro time.sleep(1) raise SmokeFailure( FailureClass.RUNTIME, - f"{label} did not become ready within {timeout_s:.0f} s ({last_err})", + # !r, like every other error this file reports: str() on an exception + # raised with no arguments is the empty string, so a readiness failure + # whose last error was e.g. a bare ConnectionRefusedError rendered as + # "did not become ready within 300 s ()" — no diagnostic at all in the + # one message a `runtime` verdict leaves behind. + f"{label} did not become ready within {timeout_s:.0f} s ({last_err!r})", ) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 02f05e6d..571b2bc7 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -14,7 +14,9 @@ import hashlib import http.client import importlib.util +import io import json +import urllib.error from pathlib import Path import pytest @@ -660,6 +662,223 @@ def test_the_script_matches_the_fake_runtime_playthrough(self) -> None: assert turn in source, f"scripted turn drifted from the e2e playthrough: {turn!r}" +# --------------------------------------------------------------------------- +# HTTP transport: status code -> failure class +# --------------------------------------------------------------------------- + + +class _Response: + """Minimum of an http.client.HTTPResponse that _request_json touches.""" + + def __init__(self, status: int = 200, body: bytes = b"{}") -> None: + self.status = status + self._body = body + + def read(self) -> bytes: + return self._body + + def __enter__(self) -> "_Response": + return self + + def __exit__(self, *a: object) -> bool: + return False + + +def _http_error(code: int, body: bytes = b"server traceback"): + def _raise(req, timeout=None): + raise urllib.error.HTTPError( + getattr(req, "full_url", "http://x"), code, "err", {}, io.BytesIO(body) + ) + + return _raise + + +class TestRequestClassification: + """`runtime` means the server broke; `pipeline` means it rejected the ask. + + Every orchestration test below replaces _request_json wholesale, so without + these the one function that decides "crashed server" vs "end-to-end + assertion" — the distinction #457 asks CI output to make — is never run. + """ + + @pytest.mark.parametrize("code", [500, 502, 503]) + def test_a_5xx_is_a_runtime_failure( + self, code: int, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr(smoke.urllib.request, "urlopen", _http_error(code)) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._request_json("http://127.0.0.1:1/api/sessions", payload={}, timeout=1) + assert exc_info.value.failure_class == smoke.FailureClass.RUNTIME + assert exc_info.value.exit_code == 4 + + @pytest.mark.parametrize("code", [400, 404, 409, 422]) + def test_a_4xx_is_a_pipeline_failure( + self, code: int, monkeypatch: pytest.MonkeyPatch + ) -> None: + # The server is up and routing; it refused what the smoke asked for — + # a 409 from POST /end on an already-ended session, say. That is an + # assertion about the API contract, not a crash. + monkeypatch.setattr(smoke.urllib.request, "urlopen", _http_error(code)) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._request_json("http://127.0.0.1:1/api/sessions", payload={}, timeout=1) + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + assert exc_info.value.exit_code == 5 + + def test_the_server_side_detail_reaches_the_failure_message( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + # The 5xx body is the only place convsim-core's own error text appears + # on the client side, and it is what the runtime remedy sends a triager + # to read. + monkeypatch.setattr( + smoke.urllib.request, "urlopen", + _http_error(500, b"Debrief generation failed"), + ) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._request_json("http://127.0.0.1:1/api/x", payload={}, timeout=1) + assert "Debrief generation failed" in str(exc_info.value) + + @pytest.mark.parametrize("exc", [ + ConnectionResetError("peer hung up"), + TimeoutError("read timed out"), + ]) + def test_a_transport_error_is_a_runtime_failure( + self, exc: BaseException, monkeypatch: pytest.MonkeyPatch + ) -> None: + def _raise(req, timeout=None): + raise exc + + monkeypatch.setattr(smoke.urllib.request, "urlopen", _raise) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._request_json("http://127.0.0.1:1/api/health", timeout=1) + assert exc_info.value.failure_class == smoke.FailureClass.RUNTIME + + def test_an_unparseable_body_is_a_runtime_failure( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + smoke.urllib.request, "urlopen", + lambda req, timeout=None: _Response(200, b"502 Bad Gateway"), + ) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._request_json("http://127.0.0.1:1/api/health", timeout=1) + assert exc_info.value.failure_class == smoke.FailureClass.RUNTIME + + def test_an_unexpected_success_status_is_a_pipeline_failure( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + # urllib raises for >= 400, so this branch only ever sees a 2xx/3xx the + # contract did not promise. + monkeypatch.setattr( + smoke.urllib.request, "urlopen", lambda req, timeout=None: _Response(204) + ) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._request_json("http://127.0.0.1:1/api/health", timeout=1) + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + assert "HTTP 204" in str(exc_info.value) + + def test_a_healthy_response_is_returned(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + smoke.urllib.request, "urlopen", + lambda req, timeout=None: _Response(200, b'{"session_id": "sess-1"}'), + ) + assert smoke._request_json( + "http://127.0.0.1:1/api/sessions", payload={"a": 1}, timeout=1 + ) == {"session_id": "sess-1"} + + +# --------------------------------------------------------------------------- +# Readiness polling +# --------------------------------------------------------------------------- + + +class _PolledProc: + def __init__(self, returncode: int | None = None) -> None: + self.returncode = returncode + + def poll(self) -> int | None: + return self.returncode + + +class TestWaitForHttp: + """A server that died on startup is nameable immediately, not after a wait.""" + + def test_a_child_that_exited_short_circuits_the_wait( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + # The realistic case is a runner OOM on model load. Polling a port + # nobody is listening on for the full timeout would bury the exit code + # and burn the wall-clock budget that exit 6 is measured against. + def _refused(*a: object, **k: object): + raise ConnectionRefusedError() + + monkeypatch.setattr(smoke.urllib.request, "urlopen", _refused) + # The timeout is deliberately far below the 300 s the real model-load + # wait allows: a regression that drops the short-circuit must turn this + # test red quickly rather than stall the per-PR suite for five minutes. + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._wait_for_http( + "http://127.0.0.1:1/v1/models", timeout_s=2.0, + label="llama-server", proc=_PolledProc(137), + ) + assert exc_info.value.failure_class == smoke.FailureClass.RUNTIME + assert "llama-server exited with code 137" in str(exc_info.value) + + def test_a_server_that_never_answers_is_a_runtime_failure( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + def _refused(*a: object, **k: object): + raise ConnectionRefusedError() + + monkeypatch.setattr(smoke.urllib.request, "urlopen", _refused) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._wait_for_http( + "http://127.0.0.1:1/api/health", timeout_s=0.0, label="convsim-core" + ) + assert exc_info.value.failure_class == smoke.FailureClass.RUNTIME + assert "convsim-core" in str(exc_info.value) + + def test_the_not_ready_message_carries_the_last_connection_error( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + # It is the only diagnostic a readiness `runtime` verdict leaves behind, + # and str() on an exception raised with no arguments is empty. + def _refused(*a: object, **k: object): + raise ConnectionRefusedError() + + monkeypatch.setattr(smoke.urllib.request, "urlopen", _refused) + # The poll backs off a second between attempts; skip the wait so this + # test costs nothing. The timeout still has to leave room for one + # attempt, or last_err is never set. + monkeypatch.setattr(smoke.time, "sleep", lambda _s: None) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._wait_for_http( + "http://127.0.0.1:1/api/health", timeout_s=0.01, label="convsim-core" + ) + assert "ConnectionRefusedError" in str(exc_info.value) + + def test_a_live_child_is_not_mistaken_for_a_crashed_one( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + smoke.urllib.request, "urlopen", lambda *a, **k: _Response(200) + ) + assert smoke._wait_for_http( + "http://127.0.0.1:1/v1/models", timeout_s=30.0, + label="llama-server", proc=_PolledProc(None), + ) is None + + def test_an_http_error_still_means_the_server_is_up( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + # A 404 is a routed response: the process is listening, which is all + # readiness means. + monkeypatch.setattr(smoke.urllib.request, "urlopen", _http_error(404)) + assert smoke._wait_for_http( + "http://127.0.0.1:1/api/health", timeout_s=30.0, label="convsim-core" + ) is None + + # --------------------------------------------------------------------------- # Full-run orchestration, with llama-server and convsim-core faked out # --------------------------------------------------------------------------- From 93fda9e6654c3ee603366c3b4e74e93d99c0d495 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 07:17:37 -0400 Subject: [PATCH 13/58] Say what an unscored debrief actually means before triage chases a regression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The new "scored debrief" assertion is the #457 acceptance criterion and stays as it is, but the thing it asserts on is thinner than the assertion implies: the debrief's dimension scores are accumulated purely from rubric_observations the *model* volunteers. Nothing asks it for them. The built-in behavioral_interview scenario defines no rubric, no prompt layer names any rubric dimensions, and the model's only hint is the bare rubric_observations array in the embedded output schema — whose empty list the schema accepts. The fake runtime returns [] always, which is why the per-PR playthrough asserts only that `scores` is a dict and never that it is non-empty. So the single most likely first failure of this job is an unscored debrief that is not a regression at all — and `pipeline`'s stock remedy, "inspect the per-turn used_fallback flags", would send a triager hunting a parse failure that never happened. The no-scores failure now carries its own note naming the cause and how to tell the two apart (did any earlier nightly ever score this scenario?), and docs/real-model-smoke.md — both copies — gets an "Unscored debrief" section the failure-class table links to from the exit-5 row. It says plainly that the fix for an array the model was never asked to fill belongs in the product, not in relaxing this check. No behaviour change: the assertion, its exit code and its class are untouched. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 27 ++++++++++++++++- docs/real-model-smoke.md | 27 ++++++++++++++++- scripts/nightly-model-smoke.py | 30 +++++++++++++++++-- tests/scripts/test_nightly_model_smoke.py | 12 ++++++++ 4 files changed, 91 insertions(+), 5 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index ecb0ab0a..e502db70 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -63,7 +63,7 @@ log, and a remedy printed next to it: | 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | | 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx | Read the child stderr tail printed above the banner | -| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact | +| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | | 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download or @@ -93,6 +93,31 @@ Each run writes the verdict, failure class, remedy, measured latencies and per-phase durations to the GitHub **step summary**, and uploads the full JSON report as the `model-smoke-report` artifact (30-day retention). +### Unscored debrief {#unscored-debrief} + +Exit 5 with *"Debrief has no rubric dimension scores"* is the one `pipeline` +failure whose first cause is probably **not** a regression, so it is worth +knowing before it happens. + +The debrief's dimension scores are accumulated entirely from +`rubric_observations` that the model volunteers on each NPC turn. Nothing asks +it for them: the built-in `behavioral_interview` scenario defines no rubric, no +prompt layer names any rubric dimensions, and the only hint the model gets is +the bare `rubric_observations` array in the embedded output schema — whose empty +list the schema accepts. The fake runtime always returns `[]`, which is why the +per-PR playthrough asserts only that `scores` *is* a dict. + +So an unscored debrief means one of two quite different things: + +- **The model left an unguided array empty.** Nothing regressed; a 4 B Q4 model + does this. If no nightly has ever scored this scenario, that is the cause — + fix it in the product (give the turn prompt a rubric layer, or pick a scenario + that defines one), not by relaxing the assertion. +- **A real regression,** if previous nightlies *did* score it. Then the per-turn + `used_fallback` flags in the report artifact are the right place to look. + +Check the previous nightly's `model-smoke-report` artifact to tell them apart. + --- ## Runtime budget diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index c5704be0..ee7879bb 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -59,7 +59,7 @@ log, and a remedy printed next to it: | 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | | 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx | Read the child stderr tail printed above the banner | -| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact | +| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | | 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download or @@ -89,6 +89,31 @@ Each run writes the verdict, failure class, remedy, measured latencies and per-phase durations to the GitHub **step summary**, and uploads the full JSON report as the `model-smoke-report` artifact (30-day retention). +### Unscored debrief + +Exit 5 with *"Debrief has no rubric dimension scores"* is the one `pipeline` +failure whose first cause is probably **not** a regression, so it is worth +knowing before it happens. + +The debrief's dimension scores are accumulated entirely from +`rubric_observations` that the model volunteers on each NPC turn. Nothing asks +it for them: the built-in `behavioral_interview` scenario defines no rubric, no +prompt layer names any rubric dimensions, and the only hint the model gets is +the bare `rubric_observations` array in the embedded output schema — whose empty +list the schema accepts. The fake runtime always returns `[]`, which is why the +per-PR playthrough asserts only that `scores` *is* a dict. + +So an unscored debrief means one of two quite different things: + +- **The model left an unguided array empty.** Nothing regressed; a 4 B Q4 model + does this. If no nightly has ever scored this scenario, that is the cause — + fix it in the product (give the turn prompt a rubric layer, or pick a scenario + that defines one), not by relaxing the assertion. +- **A real regression,** if previous nightlies *did* score it. Then the per-turn + `used_fallback` flags in the report artifact are the right place to look. + +Check the previous nightly's `model-smoke-report` artifact to tell them apart. + --- ## Runtime budget diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 9dc38c4a..88e674d7 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -160,6 +160,23 @@ class exit meaning # A debrief summary shorter than this is not a usable coaching summary. MIN_SUMMARY_CHARS = 20 +# Appended to the "no scores" failure. The debrief's dimension scores are +# accumulated purely from rubric_observations the *model* volunteers: the +# built-in behavioral_interview scenario defines no rubric, and no prompt layer +# names one (see packages/prompt-composer/.../layers.py — the model's only hint +# is the bare rubric_observations array in the embedded output schema, whose +# empty list the schema accepts). So an unscored debrief can mean the 4 B model +# simply left an unguided array empty rather than that anything regressed, and +# `pipeline`'s stock advice — inspect the per-turn used_fallback flags — would +# send triage looking for a parse failure that did not happen. +UNSCORED_DEBRIEF_NOTE = ( + "Before calling this a regression: no prompt layer names the rubric " + "dimensions, so the model is never asked for rubric_observations and an " + "empty array satisfies the turn schema. Check whether any recent nightly " + "ever scored this scenario (the fake runtime never does) — see the " + "'Unscored debrief' section of docs/real-model-smoke.md." +) + # How much slower the debrief generation is allowed to be than one NPC turn # before we stop calling it slow and start calling it hung. Only used to size # request timeouts (see run_smoke) — debrief latency itself is not budget-checked. @@ -689,7 +706,13 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: def evaluate_debrief(debrief: Optional[Dict[str, Any]]) -> tuple[List[str], List[str]]: - """Check that a *scored* debrief was produced. Returns (failures, warnings).""" + """Check that a *scored* debrief was produced. Returns (failures, warnings). + + "Scored" is the #457 acceptance criterion, so an unscored debrief is a + failure — but the scores come from rubric_observations nothing in the prompt + asks the model for, so the failure carries ``UNSCORED_DEBRIEF_NOTE`` to keep + that from reading as a regression. + """ failures: List[str] = [] warnings: List[str] = [] if not debrief: @@ -698,8 +721,9 @@ def evaluate_debrief(debrief: Optional[Dict[str, Any]]) -> tuple[List[str], List scores = debrief.get("scores") if not isinstance(scores, dict) or not scores: failures.append( - "Debrief has no rubric dimension scores; the model emitted no scorable " - "rubric_observations across the whole conversation" + "Debrief has no rubric dimension scores: no NPC turn carried a " + "rubric_observation with a rubric_id, so the debrief engine had nothing " + "to accumulate. " + UNSCORED_DEBRIEF_NOTE ) overall = debrief.get("overall_score") diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 571b2bc7..8872d75a 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -396,6 +396,18 @@ def test_unscored_debrief_fails(self) -> None: assert any("no rubric dimension scores" in f for f in failures) assert any("not numeric" in f for f in failures) + def test_unscored_debrief_says_it_may_not_be_a_regression(self) -> None: + # The scores come only from rubric_observations the model volunteers, + # and nothing in the prompt asks for them (the built-in scenario defines + # no rubric and no prompt layer names one), so an empty array satisfies + # the turn schema. `pipeline`'s stock advice — inspect the per-turn + # used_fallback flags — would send triage hunting a parse failure that + # never happened, so the failure has to carry its own note. + failures, _ = smoke.evaluate_debrief(_debrief(scores={})) + no_scores = next(f for f in failures if "no rubric dimension scores" in f) + assert smoke.UNSCORED_DEBRIEF_NOTE in no_scores + assert "real-model-smoke.md" in no_scores + def test_non_numeric_overall_score_fails(self) -> None: failures, _ = smoke.evaluate_debrief(_debrief(overall_score="52")) assert any("not numeric" in f for f in failures) From ae48b25383322b9c5dba5cfb4c8684684fc8d24d Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 07:31:10 -0400 Subject: [PATCH 14/58] Keep a malformed debug payload from failing a run the product passed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit _debug_flags_by_turn is documented as best-effort — "the debug drawer is a diagnostic, so a failure here must not turn a healthy smoke red" — and it honoured that for the transport (a 404 or a dead connection degrades to "the fallback check did not run") but not for the shape of the answer. The dict comprehension indexed t["turn_number"] unguarded, so a debug response whose turns key is absent-but-null, holds an entry without a turn number, or holds anything that is not an object raised KeyError/TypeError out of the helper, escaped to run_smoke's catch-all and reported an otherwise-green nightly as a harness bug with exit 5. That is the one verdict the issue's acceptance criterion cannot tolerate from a diagnostic: the model ran, the debrief scored, and the job goes red because a dev-only endpoint answered in a shape we did not expect. Treat an unreadable shape the way an unreachable endpoint is already treated: log what was skipped, return no flags, and let evaluate_turns warn that the check did not run — so an absent proof still never reads as a passed one. Four parametrised cases cover the shapes that used to raise (plus the empty body, which did not), asserting the run still exits 0 with the warning. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 19 ++++++++++-- tests/scripts/test_nightly_model_smoke.py | 38 +++++++++++++++++++++++ 2 files changed, 55 insertions(+), 2 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 88e674d7..593f40d5 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -899,14 +899,29 @@ def _debug_flags_by_turn(base: str, session_id: str, timeout: float) -> Dict[int """Fetch per-turn parse flags from the dev debug endpoint, keyed by game turn. Best-effort: the debug drawer is a diagnostic, so a failure here must not - turn a healthy smoke red. + turn a healthy smoke red. That holds for the *shape* of the answer as well + as for the transport — an entry without a ``turn_number`` would otherwise + raise a KeyError out of this helper, escape to ``run_smoke``'s catch-all and + report an otherwise-green run as a harness bug (exit 5). A payload we + cannot key by turn is the same situation as no payload at all: the fallback + check does not run, and ``evaluate_turns`` warns that it did not. """ try: debug = _request_json(f"{base}/sessions/{session_id}/debug", timeout=timeout) except SmokeFailure as exc: print(f"[smoke] Could not read debug flags ({exc}); continuing.") return {} - return {t["turn_number"]: t for t in debug.get("turns", [])} + turns = debug.get("turns") + if not isinstance(turns, list): + print(f"[smoke] Debug endpoint returned no turns list ({turns!r}); continuing.") + return {} + flags: Dict[int, Dict[str, Any]] = {} + for entry in turns: + if isinstance(entry, dict) and isinstance(entry.get("turn_number"), int): + flags[entry["turn_number"]] = entry + else: + print(f"[smoke] Ignoring unkeyable debug turn entry: {entry!r}") + return flags def run_smoke( diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 8872d75a..bed8f4ba 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -1404,6 +1404,44 @@ def _no_debug(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dic results = json.loads(report.read_text(encoding="utf-8")) assert any("fallback check did not run" in w for w in results["warnings"]) + @pytest.mark.parametrize( + "debug_body", + [ + {}, # no turns key at all + {"turns": None}, # turns present but not a list + {"turns": [{"used_fallback": False}]}, # entry that cannot be keyed + {"turns": ["not-a-dict"]}, + ], + ids=["no-turns-key", "turns-not-a-list", "entry-without-turn-number", "entry-not-a-dict"], + ) + def test_a_malformed_debug_payload_does_not_fail_an_otherwise_green_run( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, debug_body: dict, + ) -> None: + # Same contract as a 404 from the debug endpoint: it is a diagnostic, + # so a shape we cannot read must degrade to "the check did not run", + # not raise out of the helper and land in run_smoke's catch-all as a + # harness bug (exit 5) on a run where the product did nothing wrong. + models_dir, model_id, digest = staged_model + base = _fake_core(_debrief()) + + def _bad_debug(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/debug"): + return debug_body + return base(url, payload=payload, timeout=timeout, expect=expect) + + monkeypatch.setattr(smoke, "_request_json", _bad_debug) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == 0 + results = json.loads(report.read_text(encoding="utf-8")) + assert results["verdict"] == "pass" + assert any("fallback check did not run" in w for w in results["warnings"]) + # --------------------------------------------------------------------------- # CLI entry point From c09f3cd27c86a1e802bd20006f782883a2f5bf9e Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 07:52:45 -0400 Subject: [PATCH 15/58] Say which of the two unscored-debrief causes actually happened MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Debrief has no rubric dimension scores" is the failure this nightly is most likely to hit, and exit 5 alone does not say whether the model volunteered nothing to score or the debrief engine dropped what it was given. The remedies are opposites — one is a product gap to file, the other a bug to fix now — and the harness was sending the reader off to diff the previous nightly's artifact to tell them apart. The answer is already in the data the run collects: each turn response carries the validated rubric_observations the turn pipeline accepted, while the debrief engine re-derives its scores by re-parsing each NPC turn's *stored raw model output* with a plain json.loads. So observations present on the turns but absent from the debrief mean they were lost in between — output that needed repair, or that arrived pre-parsed from the runtime adapter, never reaches _parse_rubric_observations as valid JSON. Record rubric_observation_count per turn and the total as rubric_observations_seen, and let evaluate_debrief name the cause: zero gets the "nothing asks the model for them" note, above zero gets a regression verdict pointing at the debrief engine. Both still fail the run — a scored debrief is the #457 acceptance criterion — but the verdict no longer excuses a real bug or chases an imagined one. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 32 +++-- docs/real-model-smoke.md | 32 +++-- scripts/nightly-model-smoke.py | 128 +++++++++++++++--- tests/scripts/test_nightly_model_smoke.py | 109 ++++++++++++++- 4 files changed, 259 insertions(+), 42 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index e502db70..1a7b3828 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -96,8 +96,10 @@ report as the `model-smoke-report` artifact (30-day retention). ### Unscored debrief {#unscored-debrief} Exit 5 with *"Debrief has no rubric dimension scores"* is the one `pipeline` -failure whose first cause is probably **not** a regression, so it is worth -knowing before it happens. +failure whose cause is ambiguous from the class alone, so the harness resolves it +for you: each NPC turn's `rubric_observation_count` is recorded in the report +artifact, totalled as `rubric_observations_seen`, and the failure message names +whichever of the two causes below applies. The debrief's dimension scores are accumulated entirely from `rubric_observations` that the model volunteers on each NPC turn. Nothing asks @@ -109,14 +111,24 @@ per-PR playthrough asserts only that `scores` *is* a dict. So an unscored debrief means one of two quite different things: -- **The model left an unguided array empty.** Nothing regressed; a 4 B Q4 model - does this. If no nightly has ever scored this scenario, that is the cause — - fix it in the product (give the turn prompt a rubric layer, or pick a scenario - that defines one), not by relaxing the assertion. -- **A real regression,** if previous nightlies *did* score it. Then the per-turn - `used_fallback` flags in the report artifact are the right place to look. - -Check the previous nightly's `model-smoke-report` artifact to tell them apart. +- **`rubric_observations_seen` is 0 — the model left an unguided array empty.** + Nothing regressed; a 4 B Q4 model does this. Fix it in the product (give the + turn prompt a rubric layer, or play a scenario that defines one), not by + relaxing the assertion. +- **`rubric_observations_seen` is above 0 — a real regression.** The turns + returned observations and the debrief scored none of them, so the model did + its part. The debrief engine does not score the validated observations the + turn pipeline handed it: `_parse_rubric_observations` in + `services/convsim-core/.../debrief_engine.py` re-reads each NPC turn's *stored + raw model output* with a plain `json.loads`, so output that needed repair — or + that arrived pre-parsed from the runtime adapter — loses its observations on + the way to the debrief. Compare `raw_npc_output` from the `/debug` payload + against the per-turn `rubric_observation_count` in the artifact. + +Either way the run is red: *"scored debrief"* is the acceptance criterion for +[#457](https://github.com/outrightmental/ConversationSimulator/issues/457), and +the first cause is a product gap worth a tracking issue rather than a reason to +weaken the check. --- diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index ee7879bb..e74fc9f3 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -92,8 +92,10 @@ report as the `model-smoke-report` artifact (30-day retention). ### Unscored debrief Exit 5 with *"Debrief has no rubric dimension scores"* is the one `pipeline` -failure whose first cause is probably **not** a regression, so it is worth -knowing before it happens. +failure whose cause is ambiguous from the class alone, so the harness resolves it +for you: each NPC turn's `rubric_observation_count` is recorded in the report +artifact, totalled as `rubric_observations_seen`, and the failure message names +whichever of the two causes below applies. The debrief's dimension scores are accumulated entirely from `rubric_observations` that the model volunteers on each NPC turn. Nothing asks @@ -105,14 +107,24 @@ per-PR playthrough asserts only that `scores` *is* a dict. So an unscored debrief means one of two quite different things: -- **The model left an unguided array empty.** Nothing regressed; a 4 B Q4 model - does this. If no nightly has ever scored this scenario, that is the cause — - fix it in the product (give the turn prompt a rubric layer, or pick a scenario - that defines one), not by relaxing the assertion. -- **A real regression,** if previous nightlies *did* score it. Then the per-turn - `used_fallback` flags in the report artifact are the right place to look. - -Check the previous nightly's `model-smoke-report` artifact to tell them apart. +- **`rubric_observations_seen` is 0 — the model left an unguided array empty.** + Nothing regressed; a 4 B Q4 model does this. Fix it in the product (give the + turn prompt a rubric layer, or play a scenario that defines one), not by + relaxing the assertion. +- **`rubric_observations_seen` is above 0 — a real regression.** The turns + returned observations and the debrief scored none of them, so the model did + its part. The debrief engine does not score the validated observations the + turn pipeline handed it: `_parse_rubric_observations` in + `services/convsim-core/.../debrief_engine.py` re-reads each NPC turn's *stored + raw model output* with a plain `json.loads`, so output that needed repair — or + that arrived pre-parsed from the runtime adapter — loses its observations on + the way to the debrief. Compare `raw_npc_output` from the `/debug` payload + against the per-turn `rubric_observation_count` in the artifact. + +Either way the run is red: *"scored debrief"* is the acceptance criterion for +[#457](https://github.com/outrightmental/ConversationSimulator/issues/457), and +the first cause is a product gap worth a tracking issue rather than a reason to +weaken the check. --- diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 593f40d5..7201c0ed 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -169,12 +169,37 @@ class exit meaning # simply left an unguided array empty rather than that anything regressed, and # `pipeline`'s stock advice — inspect the per-turn used_fallback flags — would # send triage looking for a parse failure that did not happen. +# +# Used only when the run observed no rubric observations at all; when the turns +# *did* carry some, the cause is not ambiguous and UNSCORED_WITH_OBSERVATIONS_NOTE +# applies instead. UNSCORED_DEBRIEF_NOTE = ( - "Before calling this a regression: no prompt layer names the rubric " - "dimensions, so the model is never asked for rubric_observations and an " - "empty array satisfies the turn schema. Check whether any recent nightly " - "ever scored this scenario (the fake runtime never does) — see the " - "'Unscored debrief' section of docs/real-model-smoke.md." + "Before calling this a regression: no NPC turn volunteered a single rubric " + "observation, and nothing asks it to — no prompt layer names the rubric " + "dimensions, so an empty array satisfies the turn schema. This is the " + "product gap, not a regression: fix it by giving the turn prompt a rubric " + "layer or by playing a scenario that defines one. See the 'Unscored " + "debrief' section of docs/real-model-smoke.md." +) + +# The other cause of an unscored debrief, and the one that *is* a regression. +# The debrief engine does not score the validated observations the turn pipeline +# returned: it re-parses each NPC turn's stored raw model output with a plain +# json.loads (_parse_rubric_observations in debrief_engine.py) and reads +# rubric_observations out of that. So observations the turn accepted — because +# the adapter handed back pre-parsed structured output, or the parser repaired +# the text — are silently dropped when the stored raw text is not itself valid +# JSON. Saying so beats sending triage to diff last night's artifact. +UNSCORED_WITH_OBSERVATIONS_NOTE = ( + "This one IS a regression, and not an ambiguous one: the NPC turns returned " + "rubric observations, so the model did its part. The debrief engine scores " + "by re-parsing each NPC turn's stored raw model output rather than the " + "validated observations it was handed, so output that needed repair (or " + "arrived pre-parsed from the runtime) loses its observations on the way to " + "the debrief. Start at _parse_rubric_observations in " + "services/convsim-core/.../debrief_engine.py and compare raw_npc_output in " + "the /debug payload against the per-turn rubric_observation_count in the " + "report artifact." ) # How much slower the debrief generation is allowed to be than one NPC turn @@ -632,12 +657,34 @@ def _request_json( ) from exc -def _npc_turn_content(events: Sequence[dict]) -> str: - """Return the NPC utterance from a turn/start response's events, or ''.""" +def _npc_turn_payload(events: Sequence[dict]) -> Dict[str, Any]: + """Return the NPC event's payload from a turn/start response, or ``{}``.""" for ev in events: if ev.get("event_type") in ("npc_turn", "npc_opening"): - return ev.get("payload", {}).get("content", "") - return "" + payload = ev.get("payload") + return payload if isinstance(payload, dict) else {} + return {} + + +def _npc_turn_content(events: Sequence[dict]) -> str: + """Return the NPC utterance from a turn/start response's events, or ''.""" + content = _npc_turn_payload(events).get("content", "") + return content if isinstance(content, str) else "" + + +def _rubric_observation_count(events: Sequence[dict]) -> int: + """Count the rubric observations the model volunteered on this NPC turn. + + The turn response carries the *validated* observations, which is the only + way to tell the two causes of an unscored debrief apart: the debrief engine + re-derives its dimension scores from each NPC turn's stored raw model output + (``_parse_rubric_observations`` in debrief_engine.py), so observations that + were accepted on the turn but never scored mean the debrief engine lost + them, while no observations anywhere means the model simply never + volunteered any. See ``evaluate_debrief``. + """ + observations = _npc_turn_payload(events).get("rubric_observations") + return len(observations) if isinstance(observations, list) else 0 def _excerpt(text: str) -> str: @@ -705,13 +752,27 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: return (failures, warnings) -def evaluate_debrief(debrief: Optional[Dict[str, Any]]) -> tuple[List[str], List[str]]: +def evaluate_debrief( + debrief: Optional[Dict[str, Any]], + *, + rubric_observations_seen: Optional[int] = None, +) -> tuple[List[str], List[str]]: """Check that a *scored* debrief was produced. Returns (failures, warnings). "Scored" is the #457 acceptance criterion, so an unscored debrief is a - failure — but the scores come from rubric_observations nothing in the prompt - asks the model for, so the failure carries ``UNSCORED_DEBRIEF_NOTE`` to keep - that from reading as a regression. + failure — but on its own that failure does not say which of two very + different things happened, and the remedy differs completely. + ``rubric_observations_seen`` (how many rubric observations the NPC turns + actually returned, ``None`` when the run could not tell) resolves it: + + * **0** — the model volunteered nothing to score, which nothing asked it + to. The product gap, not a regression: ``UNSCORED_DEBRIEF_NOTE``. + * **> 0** — the turns carried observations and the debrief lost them. A real + regression with a named starting point: ``UNSCORED_WITH_OBSERVATIONS_NOTE``. + + Without that count the failure can only quote both possibilities and send + the reader to the previous nightly's artifact to tell them apart, which is + the one piece of triage this harness is meant to have already done. """ failures: List[str] = [] warnings: List[str] = [] @@ -720,11 +781,23 @@ def evaluate_debrief(debrief: Optional[Dict[str, Any]]) -> tuple[List[str], List scores = debrief.get("scores") if not isinstance(scores, dict) or not scores: - failures.append( - "Debrief has no rubric dimension scores: no NPC turn carried a " - "rubric_observation with a rubric_id, so the debrief engine had nothing " - "to accumulate. " + UNSCORED_DEBRIEF_NOTE - ) + if rubric_observations_seen: + cause = ( + f"the NPC turns returned {rubric_observations_seen} rubric " + "observation(s), so the debrief engine had something to accumulate " + "and did not. " + ) + UNSCORED_WITH_OBSERVATIONS_NOTE + elif rubric_observations_seen == 0: + cause = ( + "no NPC turn carried a rubric_observation, so the debrief engine " + "had nothing to accumulate. " + ) + UNSCORED_DEBRIEF_NOTE + else: + cause = ( + "the run could not tell whether any NPC turn carried a " + "rubric_observation. " + UNSCORED_DEBRIEF_NOTE + ) + failures.append(f"Debrief has no rubric dimension scores: {cause}") overall = debrief.get("overall_score") if not isinstance(overall, (int, float)) or isinstance(overall, bool): @@ -1107,6 +1180,12 @@ def run_smoke( "latency_ms": round(latency_ms), "npc_excerpt": _excerpt(npc_text), "ending_type": turn.get("ending_type"), + # What the model volunteered for the debrief to score. Recorded + # per turn so an unscored debrief can be attributed without + # re-running anything — see evaluate_debrief. + "rubric_observation_count": _rubric_observation_count( + turn.get("events", []) + ), }) if turn.get("ending_type"): # The turn pipeline already moved the session to 'Ended', so @@ -1164,14 +1243,23 @@ def run_smoke( "used_fallback": bool(debrief.get("used_fallback")), "latency_ms": round(debrief_ms), } + # Everything the NPC turns offered the debrief to score. An unscored + # debrief means something different depending on whether this is zero. + observations_seen = sum( + t.get("rubric_observation_count", 0) for t in results["turns"] + ) + results["rubric_observations_seen"] = observations_seen print(f"[smoke] Debrief in {debrief_ms:.0f} ms | " f"overall_score={debrief.get('overall_score')} " - f"scores={debrief.get('scores')}") + f"scores={debrief.get('scores')} " + f"rubric_observations_from_turns={observations_seen}") # ── assertions ──────────────────────────────────────────────────────── clock.enter("assertions") turn_failures, turn_warnings = evaluate_turns(results["turns"]) - debrief_failures, debrief_warnings = evaluate_debrief(debrief) + debrief_failures, debrief_warnings = evaluate_debrief( + debrief, rubric_observations_seen=observations_seen + ) results["warnings"] += turn_warnings + debrief_warnings pipeline_failures = turn_failures + debrief_failures if pipeline_failures: diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index bed8f4ba..b32cf6ed 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -408,6 +408,37 @@ def test_unscored_debrief_says_it_may_not_be_a_regression(self) -> None: assert smoke.UNSCORED_DEBRIEF_NOTE in no_scores assert "real-model-smoke.md" in no_scores + def test_no_observations_anywhere_names_the_product_gap(self) -> None: + # The run checked, and the model volunteered nothing to score: the + # failure can state that outright instead of listing both possibilities. + failures, _ = smoke.evaluate_debrief( + _debrief(scores={}), rubric_observations_seen=0 + ) + no_scores = next(f for f in failures if "no rubric dimension scores" in f) + assert "no NPC turn carried a rubric_observation" in no_scores + assert smoke.UNSCORED_DEBRIEF_NOTE in no_scores + assert smoke.UNSCORED_WITH_OBSERVATIONS_NOTE not in no_scores + + def test_observations_that_never_reached_the_debrief_are_a_regression(self) -> None: + # The opposite case, and the one the harness exists to catch: the turns + # returned observations and the debrief scored nothing, so the model did + # its part and the debrief engine dropped the result. Printing the + # "nothing asked the model for them" note here would excuse a real bug. + failures, _ = smoke.evaluate_debrief( + _debrief(scores={}), rubric_observations_seen=4 + ) + no_scores = next(f for f in failures if "no rubric dimension scores" in f) + assert "returned 4 rubric observation(s)" in no_scores + assert smoke.UNSCORED_WITH_OBSERVATIONS_NOTE in no_scores + assert smoke.UNSCORED_DEBRIEF_NOTE not in no_scores + assert "_parse_rubric_observations" in no_scores + + def test_a_scored_debrief_passes_whatever_the_turns_reported(self) -> None: + # The count only explains an *unscored* debrief; it must never fail a + # scored one (the debrief re-derives scores from stored raw output, so + # the two counts are not required to agree). + assert smoke.evaluate_debrief(_debrief(), rubric_observations_seen=0) == ([], []) + def test_non_numeric_overall_score_fails(self) -> None: failures, _ = smoke.evaluate_debrief(_debrief(overall_score="52")) assert any("not numeric" in f for f in failures) @@ -647,6 +678,32 @@ def test_npc_opening_content_is_extracted(self) -> None: def test_absent_npc_event_yields_empty_string(self) -> None: assert smoke._npc_turn_content([{"event_type": "tts_chunk", "payload": {}}]) == "" + def test_a_non_dict_payload_does_not_raise(self) -> None: + # The excerpt is reported, not asserted on, so a surprising payload shape + # must not turn an otherwise-green run into a harness bug (exit 5). + assert smoke._npc_turn_content([{"event_type": "npc_turn", "payload": None}]) == "" + assert smoke._npc_turn_content([{"event_type": "npc_turn"}]) == "" + + def test_rubric_observations_are_counted(self) -> None: + events = [{"event_type": "npc_turn", "payload": { + "content": "Go on.", + "rubric_observations": [ + {"rubric_id": "structure", "observation": "o", "score_delta": 1}, + {"rubric_id": "evidence", "observation": "o", "score_delta": -1}, + ], + }}] + assert smoke._rubric_observation_count(events) == 2 + + @pytest.mark.parametrize("payload", [ + {"content": "x"}, # key absent entirely + {"content": "x", "rubric_observations": []}, + {"content": "x", "rubric_observations": None}, + ]) + def test_absent_or_empty_rubric_observations_count_zero(self, payload: dict) -> None: + assert smoke._rubric_observation_count( + [{"event_type": "npc_turn", "payload": payload}] + ) == 0 + def test_excerpt_is_bounded_and_collapsed(self) -> None: excerpt = smoke._excerpt("a\n\n b" + "x" * 500) assert excerpt.startswith("a b") @@ -929,7 +986,18 @@ def kill(self) -> None: # pragma: no cover - only on a hung child "ending_type": None} -def _fake_core(debrief: dict, *, debug_turns: list | None = None): +def _npc_turn_with_observations(count: int) -> dict: + """An NPC turn whose payload carries ``count`` validated rubric observations.""" + return {"events": [{"event_type": "npc_turn", "payload": { + "content": "Walk me through that trade-off.", + "rubric_observations": [ + {"rubric_id": f"dim{i}", "observation": "o", "score_delta": 1} + for i in range(count) + ], + }}], "ending_type": None} + + +def _fake_core(debrief: dict, *, debug_turns: list | None = None, turn: dict | None = None): """Build a _request_json stand-in that answers convsim-core's endpoints.""" calls: list[str] = [] @@ -942,7 +1010,7 @@ def _request_json(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> if url.endswith("/start"): return _OPENING if url.endswith("/turn"): - return _NPC_TURN + return turn if turn is not None else _NPC_TURN if url.endswith("/debug"): return {"turns": debug_turns if debug_turns is not None else [ {"turn_number": n, "used_fallback": False, @@ -1081,6 +1149,43 @@ def test_unscored_debrief_is_a_pipeline_failure( # The artifact carries the advice, not just the class: the failure table # in the docs sends triage here for a `pipeline` verdict. assert results["remedy"] == smoke.REMEDIES[smoke.FailureClass.PIPELINE] + # The turns volunteered nothing to score, so the verdict says so rather + # than asking the reader to go and diff the previous nightly's artifact. + assert results["rubric_observations_seen"] == 0 + assert any(smoke.UNSCORED_DEBRIEF_NOTE in f for f in results["failures"]) + + def test_unscored_debrief_with_scorable_turns_is_named_a_regression( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + """The same exit 5, the opposite diagnosis. + + When the NPC turns did return rubric observations and the debrief still + scored nothing, the model is not the problem and the stock note + ("nothing asks the model for them") would excuse a real bug in the + debrief engine's score accumulation. + """ + models_dir, model_id, digest = staged_model + monkeypatch.setattr( + smoke, "_request_json", + _fake_core( + _debrief(scores={}, overall_score=None), + turn=_npc_turn_with_observations(2), + ), + ) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["rubric_observations_seen"] == 2 * len(smoke.SCRIPTED_PLAYER_TURNS) + assert all( + t["rubric_observation_count"] == 2 for t in results["turns"][1:] + ), "the per-turn count belongs in the artifact, not just the total" + assert any(smoke.UNSCORED_WITH_OBSERVATIONS_NOTE in f for f in results["failures"]) + assert not any(smoke.UNSCORED_DEBRIEF_NOTE in f for f in results["failures"]) def test_latency_regression_is_a_budget_failure_not_a_pipeline_one( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path From 2294b31c4eaf4b36a4bb9c02a28301b841036bf1 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 07:54:02 -0400 Subject: [PATCH 16/58] Stop crediting the e2e playthrough with coverage it does not provide per-PR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The new doc justifies running this job nightly rather than per-PR by pointing at `tests/e2e/test_scripted_playthrough.py` as "the per-PR equivalent". No workflow runs tests/e2e — only scripts/release-smoke.sh does, at release time. So the sentence told a reader that the scripted conversation is covered on every pull request when nothing covers it until a release, which is the opposite of the reassurance it was offering. Name where that playthrough actually runs, and say what does run per-PR: this harness's own unit tests. Same correction in the script comment beside SCRIPTED_PLAYER_TURNS and in the test that guards the two scripts against drift. Co-Authored-By: Claude Opus 5 --- docs-site/src/content/docs/dev/real-model-smoke.md | 10 ++++++---- docs/real-model-smoke.md | 10 ++++++---- scripts/nightly-model-smoke.py | 5 +++-- tests/scripts/test_nightly_model_smoke.py | 5 +++-- 4 files changed, 18 insertions(+), 12 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 1a7b3828..930062a0 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -26,9 +26,11 @@ on a **real local model** end-to-end — registry download → llama.cpp → | Runtime budget | **< 30 min** (`timeout-minutes: 30`); the harness self-limits to 20 min | > Nightly, not per-PR, on purpose: a 2.5 GB download plus ~11 min of CPU-only -> inference cannot sit in the PR path. The per-PR equivalent is the fake-runtime -> playthrough in `tests/e2e/test_scripted_playthrough.py`, which uses the *same -> scripted player turns* so the two cover the same conversation shape. +> inference cannot sit in the PR path. The fake-runtime counterpart is the +> playthrough in `tests/e2e/test_scripted_playthrough.py` — run at release time +> by `scripts/release-smoke.sh`, not on pull requests — which uses the *same +> scripted player turns*, so the two cover the same conversation shape against +> different runtimes. What *does* run per-PR is this harness's own unit tests. --- @@ -107,7 +109,7 @@ it for them: the built-in `behavioral_interview` scenario defines no rubric, no prompt layer names any rubric dimensions, and the only hint the model gets is the bare `rubric_observations` array in the embedded output schema — whose empty list the schema accepts. The fake runtime always returns `[]`, which is why the -per-PR playthrough asserts only that `scores` *is* a dict. +release-time playthrough asserts only that `scores` *is* a dict. So an unscored debrief means one of two quite different things: diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index e74fc9f3..b9f86fd5 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -22,9 +22,11 @@ on a **real local model** end-to-end — registry download → llama.cpp → | Runtime budget | **< 30 min** (`timeout-minutes: 30`); the harness self-limits to 20 min | > Nightly, not per-PR, on purpose: a 2.5 GB download plus ~11 min of CPU-only -> inference cannot sit in the PR path. The per-PR equivalent is the fake-runtime -> playthrough in `tests/e2e/test_scripted_playthrough.py`, which uses the *same -> scripted player turns* so the two cover the same conversation shape. +> inference cannot sit in the PR path. The fake-runtime counterpart is the +> playthrough in `tests/e2e/test_scripted_playthrough.py` — run at release time +> by `scripts/release-smoke.sh`, not on pull requests — which uses the *same +> scripted player turns*, so the two cover the same conversation shape against +> different runtimes. What *does* run per-PR is this harness's own unit tests. --- @@ -103,7 +105,7 @@ it for them: the built-in `behavioral_interview` scenario defines no rubric, no prompt layer names any rubric dimensions, and the only hint the model gets is the bare `rubric_observations` array in the embedded output schema — whose empty list the schema accepts. The fake runtime always returns `[]`, which is why the -per-PR playthrough asserts only that `scores` *is* a dict. +release-time playthrough asserts only that `scores` *is* a dict. So an unscored debrief means one of two quite different things: diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 7201c0ed..926e4f85 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -144,8 +144,9 @@ class exit meaning DEFAULT_WALL_CLOCK_BUDGET_S = 1200.0 # 20 min # Scripted player turns — deliberately the same script as the fake-runtime -# playthrough in tests/e2e/test_scripted_playthrough.py, so the real-model -# nightly and the per-PR smoke exercise the same conversation shape. +# playthrough in tests/e2e/test_scripted_playthrough.py (run at release time by +# scripts/release-smoke.sh), so the two smokes exercise the same conversation +# shape against different runtimes. SCRIPTED_PLAYER_TURNS = ( "I have five years of experience in software development.", "My biggest achievement was building a cross-team API platform.", diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index b32cf6ed..5caa9eaf 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -722,8 +722,9 @@ def test_the_script_is_a_multi_turn_conversation(self) -> None: assert all(t.strip() for t in smoke.SCRIPTED_PLAYER_TURNS) def test_the_script_matches_the_fake_runtime_playthrough(self) -> None: - # Keeping the two harnesses on the same script means the nightly - # exercises the conversation shape the per-PR smoke already covers. + # Keeping the two harnesses on the same script means the nightly and + # the release-time fake-runtime smoke cover the same conversation shape, + # so a difference between them is the runtime and nothing else. source = (REPO_ROOT / "tests" / "e2e" / "test_scripted_playthrough.py").read_text( encoding="utf-8" ) From d98cdf04f2555dd441cc3ad0c6e2b9caee748bdd Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 07:56:39 -0400 Subject: [PATCH 17/58] Stop a download verdict from reporting a wall clock it never started MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pre-run modes — the registry lookup, --download-only, --verify-only — write a step summary through the same renderer as a full run, but they keep no clock. Defaulting the missing timings to zero put "Wall clock: 0 s (budget 0 s)" at the top of every download and checksum verdict: a measurement that was never taken, in the first line a triager reads, on the two classes whose whole point is that they say nothing about how long anything took. Omit the line when there is no measurement to report. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 12 ++++++++++-- tests/scripts/test_nightly_model_smoke.py | 16 ++++++++++++++++ 2 files changed, 26 insertions(+), 2 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 926e4f85..9c739b84 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -861,9 +861,17 @@ def render_step_summary(results: Dict[str, Any]) -> str: f"## {icon} Real-model smoke: {verdict.upper()}", "", f"- **Model:** `{results.get('model_id')}`", - f"- **Wall clock:** {results.get('wall_clock_s', 0):.0f} s " - f"(budget {results.get('wall_clock_budget_s', 0):.0f} s)", ] + # Only the smoke run keeps a clock. The pre-run modes (registry lookup, + # download, verify) write a summary too, and defaulting their missing + # timings to zero printed "Wall clock: 0 s (budget 0 s)" at the top of a + # download or checksum verdict — a measurement the run never took, in the + # first thing a triager reads. + if results.get("wall_clock_s") is not None: + lines.append( + f"- **Wall clock:** {results['wall_clock_s']:.0f} s " + f"(budget {results.get('wall_clock_budget_s', 0):.0f} s)" + ) if verdict == "fail": cls = results.get("failure_class", "unknown") lines += [ diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 5caa9eaf..ab0b0730 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -638,6 +638,22 @@ def test_pass_summary_tabulates_measurements_with_ceilings(self) -> None: assert "240000 ms" in text # 10 000 × 20 × 1.2 assert "| `debrief_ms` | 200000 ms | — |" in text + def test_a_pre_run_failure_does_not_claim_a_zero_second_run(self) -> None: + # --download-only / --verify-only and the registry lookup write a summary + # without ever starting a clock; reporting "Wall clock: 0 s (budget 0 s)" + # on a download verdict invents a measurement the run never took. + text = smoke.render_step_summary({ + "verdict": "fail", + "model_id": "qwen3-4b", + "failure_class": smoke.FailureClass.DOWNLOAD, + "exit_code": 2, + "failures": ["Download failed for https://…: TimeoutError()"], + "remedy": smoke.REMEDIES[smoke.FailureClass.DOWNLOAD], + }) + assert "Wall clock" not in text + assert "`download`" in text + assert smoke.REMEDIES[smoke.FailureClass.DOWNLOAD] in text + def test_summary_is_not_written_without_the_github_env_var( self, monkeypatch: pytest.MonkeyPatch ) -> None: From 70a30248a92049a1145604ca050f7e9d1599f691 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 08:07:10 -0400 Subject: [PATCH 18/58] Point the performance doc at the budget the nightly actually enforces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The latency table lists cold start, TTFT, TTS first-audio and STT round-trip, and the sentence above it now claims the nightly real-model smoke "flags any regression greater than 20 % against these values". It enforces none of them: cold start, TTS and STT are never exercised by that job, and TTFT is explicitly excluded by the harness because it needs the token-streaming gateway the smoke does not stand up. The one budget it does enforce — FULL_RESPONSE_MS, 10 s — was absent from the table entirely, even though the closing line promises the table mirrors LATENCY_BUDGETS. So linking the job made a false claim more specific and more credible instead of giving it a referent. Add the missing full-NPC-response row (the table now matches all five LATENCY_BUDGETS entries) and scope the sentence to it, saying plainly which budgets are hand-verified and why. Two tests keep it from drifting again: BUDGETS_MS is a hand-maintained copy of the TypeScript constant, so one asserts they still agree — a nightly failing against a stale budget is worse than no nightly — and the other asserts both copies of the performance doc still publish the figure the job scales, so the CI-coverage claim keeps a referent a reader can check. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/play/performance.md | 3 +- docs/performance.md | 3 +- tests/scripts/test_nightly_model_smoke.py | 41 +++++++++++++++++++ 3 files changed, 45 insertions(+), 2 deletions(-) diff --git a/docs-site/src/content/docs/play/performance.md b/docs-site/src/content/docs/play/performance.md index 3a443203..d8bb9fc7 100644 --- a/docs-site/src/content/docs/play/performance.md +++ b/docs-site/src/content/docs/play/performance.md @@ -9,12 +9,13 @@ ConversationSimulator runs a local LLM, optionally local STT (Whisper) and TTS ( ## Latency budgets -These are the official latency budgets for the **mid-spec reference machine** (Apple M2 / NVIDIA RTX 3060 equivalent). In-app performance warnings fire when a measurement exceeds the corresponding budget. The [nightly real-model smoke test](/dev/real-model-smoke/) flags any regression greater than 20 % against these values, scaled for CPU-only CI hardware. +These are the official latency budgets for the **mid-spec reference machine** (Apple M2 / NVIDIA RTX 3060 equivalent). In-app performance warnings fire when a measurement exceeds the corresponding budget. Of these, only **full NPC response** is enforced automatically: the [nightly real-model smoke test](/dev/real-model-smoke/) scales that budget for CPU-only CI hardware and fails when a measurement exceeds the scaled budget by more than 20 %. The others are verified by hand — cold start, TTS and STT are not exercised by that job, and TTFT needs the token-streaming gateway it does not start. | Metric | Budget | Condition | |--------|--------|-----------| | Cold start → interactive Home | < 10 s | Model already downloaded; startup to playable state | | Time-to-first-token (TTFT) | < 2.5 s | Starter model (4 B Q4\_K\_M) on recommended-tier hardware | +| Full NPC response | < 10 s | Starter model, complete reply returned; the one budget the nightly CI smoke enforces | | TTS first-audio chunk | < 1.5 s | Kokoro sidecar, sentence-level streaming | | STT round-trip | < 2 s | 10-word utterance, whisper.cpp small model | diff --git a/docs/performance.md b/docs/performance.md index 5b07f168..db29f052 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -4,12 +4,13 @@ ConversationSimulator runs a local LLM, optionally local STT (Whisper) and TTS ( ## Latency budgets -These are the official latency budgets for the **mid-spec reference machine** (Apple M2 / NVIDIA RTX 3060 equivalent). In-app performance warnings fire when a measurement exceeds the corresponding budget. The [nightly real-model smoke test](real-model-smoke.md) flags any regression greater than 20 % against these values, scaled for CPU-only CI hardware. +These are the official latency budgets for the **mid-spec reference machine** (Apple M2 / NVIDIA RTX 3060 equivalent). In-app performance warnings fire when a measurement exceeds the corresponding budget. Of these, only **full NPC response** is enforced automatically: the [nightly real-model smoke test](real-model-smoke.md) scales that budget for CPU-only CI hardware and fails when a measurement exceeds the scaled budget by more than 20 %. The others are verified by hand — cold start, TTS and STT are not exercised by that job, and TTFT needs the token-streaming gateway it does not start. | Metric | Budget | Condition | |--------|--------|-----------| | Cold start → interactive Home | < 10 s | Model already downloaded; startup to playable state | | Time-to-first-token (TTFT) | < 2.5 s | Starter model (4 B Q4\_K\_M) on recommended-tier hardware | +| Full NPC response | < 10 s | Starter model, complete reply returned; the one budget the nightly CI smoke enforces | | TTS first-audio chunk | < 1.5 s | Kokoro sidecar, sentence-level streaming | | STT round-trip | < 2 s | 10-word utterance, whisper.cpp small model | diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index ab0b0730..2332fb6b 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -502,6 +502,47 @@ def test_failure_message_shows_the_arithmetic(self) -> None: assert "documented budget 10000 ms × 20.0 × 1.2" in failures[0] +class TestDocumentedBudgetAgreement: + """BUDGETS_MS must agree with the budget the product publishes. + + The harness deliberately does not import the TypeScript constant, so the + only thing keeping the number it enforces equal to the number the product + promises is a hand-maintained copy. These tests make that copy load-bearing: + a nightly that fails against a stale budget is worse than no nightly, and a + performance doc that advertises CI coverage of a budget it no longer lists + leaves the claim with no referent. + """ + + def test_the_shared_latency_constant_agrees_with_the_harness(self) -> None: + import re + + metrics = ( + REPO_ROOT / "packages" / "shared" / "src" / "types" / "metrics.ts" + ).read_text(encoding="utf-8") + match = re.search(r"FULL_RESPONSE_MS:\s*([\d_]+)", metrics) + assert match, "LATENCY_BUDGETS.FULL_RESPONSE_MS not found in metrics.ts" + assert int(match.group(1).replace("_", "")) == smoke.BUDGETS_MS["full_response_ms"] + + @pytest.mark.parametrize( + "doc_path", + [ + ("docs", "performance.md"), + ("docs-site", "src", "content", "docs", "play", "performance.md"), + ], + ) + def test_the_enforced_budget_is_published_in_the_performance_doc( + self, doc_path: tuple + ) -> None: + doc = REPO_ROOT.joinpath(*doc_path).read_text(encoding="utf-8") + seconds = smoke.BUDGETS_MS["full_response_ms"] / 1000 + assert "Full NPC response" in doc, ( + "the only budget this job enforces is missing from the latency table, " + "so the doc's claim about CI coverage names nothing a reader can check" + ) + assert f"< {seconds:.0f} s" in doc + assert "real-model-smoke" in doc + + # --------------------------------------------------------------------------- # Wall-clock budget # --------------------------------------------------------------------------- From 8ab6ef998e101c6c44a431a6b0bbbb18495b9622 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 08:17:12 -0400 Subject: [PATCH 19/58] Say when the nightly played one turn instead of three MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The NPC can close the conversation itself — session_control.continue_session — and the harness correctly treats that as a pass: the product handled an early ending the way it is designed to, and the turns that did run were still real model output. But the job exists to play a *multi-turn* conversation, and a run that played one turn proves less than one that played three. Nothing said so: the step summary showed a green verdict, and only the per-turn array in the JSON artifact (or the log line nobody reads on a green run) revealed the conversation had been cut short. That is the one place this harness implied a proof it did not have, having been careful everywhere else — a missing parse flag warns, an unverified checksum warns. Record scripted_turns_played and warn with the ending_type when it is short of the script, so a green run on one turn cannot be mistaken for a green run on three. Documented in both copies of the real-model-smoke doc, and both the short and the full playthrough are now covered by orchestration tests. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 6 ++++++ docs/real-model-smoke.md | 6 ++++++ scripts/nightly-model-smoke.py | 15 +++++++++++++ tests/scripts/test_nightly_model_smoke.py | 21 +++++++++++++++++++ 4 files changed, 48 insertions(+) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 930062a0..9ab75ae5 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -51,6 +51,12 @@ on a **real local model** end-to-end — registry download → llama.cpp → The NPC *opening* line is authored scenario text, not a generation — so the scripted player turns, not the opening, are what prove the model is working. +If the NPC closes the conversation before the script runs out +(`session_control.continue_session`), the run still passes on the turns it did +play — that is the product working as designed — but it records +`scripted_turns_played` and warns in the step summary, so a green run on one +turn is not mistaken for a green run on three. + --- ## Failure classification diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index b9f86fd5..78d0f494 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -47,6 +47,12 @@ on a **real local model** end-to-end — registry download → llama.cpp → The NPC *opening* line is authored scenario text, not a generation — so the scripted player turns, not the opening, are what prove the model is working. +If the NPC closes the conversation before the script runs out +(`session_control.continue_session`), the run still passes on the turns it did +play — that is the product working as designed — but it records +`scripted_turns_played` and warns in the step summary, so a green run on one +turn is not mistaken for a green run on three. + --- ## Failure classification diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 9c739b84..79978505 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1205,6 +1205,21 @@ def run_smoke( "stopping the scripted turns.") break + results["scripted_turns_played"] = len(turn_latencies) + if len(turn_latencies) < len(SCRIPTED_PLAYER_TURNS): + # Not a failure: the model is allowed to close the conversation + # (session_control.continue_session), and the product handled it. + # But the job exists to play a *multi-turn* conversation, and a run + # that played one turn proves less than one that played three — say + # so in the summary rather than let a short green run read like a + # full one. + results["warnings"].append( + f"Only {len(turn_latencies)} of {len(SCRIPTED_PLAYER_TURNS)} scripted " + f"player turns ran: the NPC ended the session early (ending_type=" + f"{results['turns'][-1].get('ending_type')!r}), so this run proves " + "less about multi-turn behaviour than a full playthrough" + ) + # The median is the headline full-response figure: a single unlucky # turn (runner steal, a long NPC answer) must not flap the nightly, # while a real regression moves every turn and so moves the median. diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 2332fb6b..e03c166f 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -1540,6 +1540,27 @@ def _ends_after_one(url: str, *, payload=None, timeout=None, expect=(200, 201)) "must not POST /end to a session the scenario already ended" ) assert results["debrief"] is not None + # Green, but on one turn instead of three. The job exists to play a + # multi-turn conversation, so a short run must not read like a full one. + assert results["scripted_turns_played"] == 1 + assert any( + f"Only 1 of {len(smoke.SCRIPTED_PLAYER_TURNS)} scripted player turns ran" in w + for w in results["warnings"] + ), results["warnings"] + + def test_a_full_playthrough_does_not_warn_about_a_short_run( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core(_debrief())) + report = tmp_path / "report.json" + + assert smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) == 0 + results = json.loads(report.read_text(encoding="utf-8")) + assert results["scripted_turns_played"] == len(smoke.SCRIPTED_PLAYER_TURNS) + assert results["warnings"] == [] def test_unavailable_debug_flags_warn_instead_of_silently_passing( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path From 7927da3fbe0a859aca7a4f670c655a25383e2f39 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 17:07:35 -0400 Subject: [PATCH 20/58] Say when a timed-out nightly is really a latency regression Three scripted turns at the 240 s CI ceiling plus a debrief allowed twice that very nearly fill the 20 min wall-clock budget, so a uniform ~2x slowdown -- exactly the regression the `budget` class exists to catch -- exhausts the clock before the budget phase ever runs. Since `timeout` outranks `budget`, exit 1 was effectively unreachable for the failure it names, and exit 6's remedy ("inference got much slower or a phase hung") left triage to work out which of the two had happened. The class stays `timeout` -- the clock really did run out -- but when the NPC turns that did complete already have a median past the CI ceiling, the verdict now appends that measurement and says it is most likely a latency regression rather than a hang. The authored opening is excluded (it is replayed text, not inference) and the median is used rather than the worst turn, so one unlucky turn cannot invent the evidence. Tests cover the new helper directly and the end-to-end case where a run times out during the debrief after measurably slow turns, plus a previously untested headline behaviour the fix leans on: full_response_ms is the median of the scripted turns, not their worst. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 9 ++ docs/real-model-smoke.md | 9 ++ scripts/nightly-model-smoke.py | 52 +++++++ tests/scripts/test_nightly_model_smoke.py | 135 ++++++++++++++++++ 4 files changed, 205 insertions(+) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 9ab75ae5..dcbbbaf5 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -97,6 +97,15 @@ rather than the symptom the client happened to see: client-side and *looks* like an unresponsive server. It is reported as `timeout`, whose remedy points at `phase_durations_s`. +Because `timeout` outranks `budget`, a *uniform* slowdown normally surfaces as +exit 6 rather than exit 1: three turns at the 240 s CI ceiling plus a debrief +allowed twice that very nearly fill the 20 min budget, so a ~2x regression +exhausts the clock before the budget phase runs. The class stays `timeout` — the +clock really did run out — but when the turns that *did* complete already have a +median past the ceiling, the verdict says so and names the measurement, so exit 6 +is not mistaken for a hang. Exit 1 remains the verdict when the conversation +finishes inside the budget but too slowly. + Each run writes the verdict, failure class, remedy, measured latencies and per-phase durations to the GitHub **step summary**, and uploads the full JSON report as the `model-smoke-report` artifact (30-day retention). diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 78d0f494..9ca49cea 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -93,6 +93,15 @@ rather than the symptom the client happened to see: client-side and *looks* like an unresponsive server. It is reported as `timeout`, whose remedy points at `phase_durations_s`. +Because `timeout` outranks `budget`, a *uniform* slowdown normally surfaces as +exit 6 rather than exit 1: three turns at the 240 s CI ceiling plus a debrief +allowed twice that very nearly fill the 20 min budget, so a ~2x regression +exhausts the clock before the budget phase runs. The class stays `timeout` — the +clock really did run out — but when the turns that *did* complete already have a +median past the ceiling, the verdict says so and names the measurement, so exit 6 +is not mistaken for a hang. Exit 1 remains the verdict when the conversation +finishes inside the budget but too slowly. + Each run writes the verdict, failure class, remedy, measured latencies and per-phase durations to the GitHub **step summary**, and uploads the full JSON report as the `model-smoke-report` artifact (30-day retention). diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 79978505..bb28526f 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -55,6 +55,13 @@ class exit meaning failures carry their own remedy instead of ``pipeline``'s, because "inspect the per-turn used_fallback flags" is the wrong first move for any of them. +``timeout`` outranks ``budget``, and the two thresholds are close together by +construction (three turns at the CI ceiling plus a debrief very nearly fill the +wall-clock budget), so a uniform latency regression usually exhausts the clock +before the budget phase runs. The class stays ``timeout``, but the verdict then +names the measured turn latencies that say it was a regression rather than a +hang — see ``timeout_latency_evidence``. + Transcript content is fully scripted (see ``SCRIPTED_PLAYER_TURNS``) and holds no user data, so short NPC excerpts are logged and reported — they are the evidence that a real model, not a canned fallback, drove the conversation. @@ -850,6 +857,44 @@ def evaluate_budgets( return (failures, lines) +def timeout_latency_evidence( + turns: Sequence[Dict[str, Any]], ceiling_ms: float +) -> Optional[str]: + """Name the latency regression hiding inside a ``timeout`` verdict, if there is one. + + The wall-clock budget and the latency ceiling sit close together by + construction: three scripted turns at the CI ceiling (240 s each) plus a + debrief allowed twice that very nearly fill the 20 min budget. So a uniform + slowdown large enough to fail the budget check usually exhausts the clock + *before* the budget phase runs, and TIMEOUT — which outranks everything but + a crashed child — is what gets reported. That class is honest, the clock + did run out, but its remedy ("inference got much slower or a phase hung") + leaves the reader to work out which. When the turns that *did* complete are + already past the ceiling, we know which, so say so. + + Returns ``None`` when the completed turns were inside the ceiling (a genuine + hang, or a phase other than the conversation ran long) or when no + model-generated turn finished at all. + """ + latencies = [ + float(t["latency_ms"]) + for t in turns + if t.get("model_generated") and isinstance(t.get("latency_ms"), (int, float)) + ] + if not latencies: + return None + median_ms = statistics.median(latencies) + if median_ms <= ceiling_ms: + return None + return ( + f"The {len(latencies)} NPC turn(s) that did complete have a median latency of " + f"{median_ms:.0f} ms, already past the {ceiling_ms:.0f} ms CI ceiling: this is " + "most likely the latency regression `budget` exists to catch rather than a " + "hung phase. Compare measured_ms in the report artifact against recent " + "nightlies before raising --wall-clock-budget-s." + ) + + # ── Reporting ───────────────────────────────────────────────────────────────── @@ -1351,6 +1396,13 @@ def run_smoke( f"Wall-clock budget of {wall_clock_budget_s:.0f} s exhausted during phase " f"{results['failed_phase']!r} (elapsed {clock.elapsed_s:.0f} s)", ) + if failure_class == FailureClass.TIMEOUT: + # TIMEOUT is reached before the budget phase can run for exactly the + # regression `budget` is meant to catch, so attach the measurement + # that tells the two apart. See timeout_latency_evidence. + evidence = timeout_latency_evidence(results["turns"], full_ceiling_s * 1000) + if evidence: + results["failures"].append(evidence) _dump_stderr_tails((("convsim-core", core_stderr_tail), ("llama-server", llama_stderr_tail))) _print_banner(failure_class, results["failures"], remedy=results["remedy"]) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index e03c166f..28b7815d 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -16,6 +16,7 @@ import importlib.util import io import json +import statistics import urllib.error from pathlib import Path @@ -502,6 +503,61 @@ def test_failure_message_shows_the_arithmetic(self) -> None: assert "documented budget 10000 ms × 20.0 × 1.2" in failures[0] +class TestTimeoutLatencyEvidence: + """A `timeout` verdict must say when the real story is a latency regression. + + Three turns at the 240 s CI ceiling plus a debrief allowed twice that very + nearly fill the 20 min wall-clock budget, so a uniform ~2x slowdown — the + regression `budget` exists to catch — exhausts the clock before the budget + phase ever runs. TIMEOUT outranks it, so the measurement has to be attached + to the verdict or the remedy ("slower, or hung?") leaves triage guessing. + """ + + @staticmethod + def _turns(*latencies_ms: float) -> list[dict]: + return [ + {"label": "npc_opening", "turn_number": 0, "model_generated": False, + "latency_ms": 10}, + *[ + {"label": f"player_turn_{i}", "turn_number": i, + "model_generated": True, "latency_ms": ms} + for i, ms in enumerate(latencies_ms, start=1) + ], + ] + + def test_turns_past_the_ceiling_are_named_as_a_likely_regression(self) -> None: + evidence = smoke.timeout_latency_evidence( + self._turns(250_000, 260_000, 270_000), 240_000 + ) + assert evidence is not None + assert "260000 ms" in evidence # the median of the completed turns + assert "240000 ms CI ceiling" in evidence + assert "`budget`" in evidence + + def test_turns_inside_the_ceiling_stay_a_plain_timeout(self) -> None: + # Something hung, or a phase other than the conversation ran long: + # claiming a latency regression here would be a guess. + assert smoke.timeout_latency_evidence( + self._turns(10_000, 11_000, 12_000), 240_000 + ) is None + + def test_a_run_that_timed_out_before_any_turn_says_nothing(self) -> None: + assert smoke.timeout_latency_evidence(self._turns(), 240_000) is None + + def test_the_authored_opening_is_not_evidence_of_slow_inference(self) -> None: + # The opening is replayed scenario text, so its latency says nothing + # about the model -- counting it would invent evidence. + slow_opening = [{"label": "npc_opening", "turn_number": 0, + "model_generated": False, "latency_ms": 10 ** 9}] + assert smoke.timeout_latency_evidence(slow_opening, 240_000) is None + + def test_one_slow_turn_among_fast_ones_is_not_called_a_regression(self) -> None: + # Same reason full_response_ms is a median: one unlucky turn is noise. + assert smoke.timeout_latency_evidence( + self._turns(10_000, 10 ** 9, 12_000), 240_000 + ) is None + + class TestDocumentedBudgetAgreement: """BUDGETS_MS must agree with the budget the product publishes. @@ -1271,6 +1327,85 @@ def _slow_turn(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> di assert results["failure_class"] == smoke.FailureClass.BUDGET assert results["failed_phase"] == "budget" + def test_the_headline_full_response_is_the_median_not_the_worst_turn( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # One unlucky turn -- runner steal, an unusually long NPC answer -- must + # not move the figure the budget judges, or the nightly flaps. A real + # regression moves every turn and so moves the median. + import time + + models_dir, model_id, digest = staged_model + base = _fake_core(_debrief()) + total_turns = len(smoke.SCRIPTED_PLAYER_TURNS) + seen = {"turns": 0} + + def _one_slow_turn(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/turn"): + seen["turns"] += 1 + if seen["turns"] == total_turns: # only the last turn is slow + time.sleep(0.25) + return base(url, payload=payload, timeout=timeout, expect=expect) + + monkeypatch.setattr(smoke, "_request_json", _one_slow_turn) + report = tmp_path / "report.json" + + assert smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) == 0 + + results = json.loads(report.read_text(encoding="utf-8")) + per_turn = sorted( + t["latency_ms"] for t in results["turns"] if t["model_generated"] + ) + assert len(per_turn) == total_turns + headline = results["measured_ms"]["full_response_ms"] + # The median of the turns, not their worst and not their mean. + assert abs(headline - statistics.median(per_turn)) <= 1 + assert results["measured_ms"]["full_response_max_ms"] == per_turn[-1] + # The slow turn is still recorded -- it is just not the headline. + assert per_turn[-1] - headline > 100 + + def test_a_timeout_names_the_latency_regression_it_hides( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # Three turns at the CI ceiling plus a debrief allowed twice that very + # nearly fill the wall-clock budget, so the regression `budget` exists to + # catch exhausts the clock before the budget phase runs. TIMEOUT is the + # honest class, but it has to carry the measurement or its remedy + # ("slower, or hung?") leaves triage to guess. + import time + + models_dir, model_id, digest = staged_model + base = _fake_core(_debrief()) + + def _slow(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/turn"): + time.sleep(0.05) # measurable, so a 0 ms ceiling is exceeded + if url.endswith("/debrief"): + time.sleep(0.6) # outlast the budget below + raise smoke.SmokeFailure( + smoke.FailureClass.RUNTIME, f"{url} did not answer with usable JSON" + ) + return base(url, payload=payload, timeout=timeout, expect=expect) + + monkeypatch.setattr(smoke, "_request_json", _slow) + report = tmp_path / "report.json" + + # Factor 0 => a 0 ms ceiling, so the measured turns are over it. + exit_code = smoke.run_smoke( + model_id, 0.0, report, model_sha256=digest, models_dir=models_dir, + wall_clock_budget_s=0.5, + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.TIMEOUT] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failure_class"] == smoke.FailureClass.TIMEOUT + assert any( + "latency regression" in f and "CI ceiling" in f + for f in results["failures"] + ) + def test_a_crashed_child_outranks_the_client_side_symptom( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: From 6baa2b1ce1d5856f37369e81713a62c1cc1d8d6f Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 18:41:02 -0400 Subject: [PATCH 21/58] Stop the unscored-debrief note from closing the investigation it should open MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified the harness end-to-end against the real starter model (Qwen3 4B Q4_K_M under the turn-output schema, via llama-server + convsim-core with CONVSIM_RUNTIME_ID=llama_cpp): three model-generated turns with used_fallback=false and used_native_structured_output=true on every turn, and a scored debrief — three to four rubric dimensions, overall_score near 50, a 269-character summary, no fallback narrative. Across two runs, all six scripted turns volunteered at least one rubric observation. That contradicts what UNSCORED_DEBRIEF_NOTE and the doc told a triager. Both said a zero-observation debrief is "the product gap, not a regression" and sent the reader off to file a prompt-layer issue. The missing rubric prompt layer is real and is what makes the failure reachable, but an empty array is not the resting state: a whole conversation producing none means something changed about what reaches the model — the OUTPUT_SCHEMA layer, the registry's starter pin, the adapter's JSON-schema constraint, sampling settings — which is exactly the regression class this nightly exists to catch. Writing it off in the failure text would have closed that investigation before it started. The note and the docs/real-model-smoke.md section now keep the thin prompt coverage as context and name the things to check first, and record the measured behaviour so the next reader knows what normal looks like. Renames the two tests that encoded the old framing and makes the correction load-bearing: the note must still tell triage not to write the run off. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 27 ++++++++--- docs/real-model-smoke.md | 27 ++++++++--- scripts/nightly-model-smoke.py | 47 +++++++++++++------ tests/scripts/test_nightly_model_smoke.py | 11 ++++- 4 files changed, 81 insertions(+), 31 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index dcbbbaf5..38b24aa0 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -126,12 +126,24 @@ the bare `rubric_observations` array in the embedded output schema — whose emp list the schema accepts. The fake runtime always returns `[]`, which is why the release-time playthrough asserts only that `scores` *is* a dict. +That thin prompt coverage is the standing weakness that makes an unscored +debrief reachable at all. It is *not*, however, the expected outcome: in local +verification against the real starter model — Qwen3 4B Q4\_K\_M, the same pin +the nightly uses, under the turn-output schema — every one of six scripted turns +across two runs volunteered at least one observation, and the debrief scored +three to four dimensions with an `overall_score` near 50. Treat zero +observations as a signal, not as the resting state. + So an unscored debrief means one of two quite different things: -- **`rubric_observations_seen` is 0 — the model left an unguided array empty.** - Nothing regressed; a 4 B Q4 model does this. Fix it in the product (give the - turn prompt a rubric layer, or play a scenario that defines one), not by - relaxing the assertion. +- **`rubric_observations_seen` is 0 — the model volunteered nothing.** Only the + prompt's bare schema hint was ever asking, so this is reachable by design — + but since the starter model does normally answer that hint, first check what + changed about what reaches the model: the `OUTPUT_SCHEMA` prompt layer, the + registry's `starter` pin, the adapter's JSON-schema constraint, sampling + settings. Then fix the weakness itself in the product (give the turn prompt a + rubric layer, or play a scenario that defines one), not by relaxing the + assertion. - **`rubric_observations_seen` is above 0 — a real regression.** The turns returned observations and the debrief scored none of them, so the model did its part. The debrief engine does not score the validated observations the @@ -143,9 +155,10 @@ So an unscored debrief means one of two quite different things: against the per-turn `rubric_observation_count` in the artifact. Either way the run is red: *"scored debrief"* is the acceptance criterion for -[#457](https://github.com/outrightmental/ConversationSimulator/issues/457), and -the first cause is a product gap worth a tracking issue rather than a reason to -weaken the check. +[#457](https://github.com/outrightmental/ConversationSimulator/issues/457). +Neither cause is a reason to weaken the check — the first is worth a tracking +issue for the missing rubric prompt layer *and* an investigation of the run that +hit it. --- diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 9ca49cea..8b465687 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -122,12 +122,24 @@ the bare `rubric_observations` array in the embedded output schema — whose emp list the schema accepts. The fake runtime always returns `[]`, which is why the release-time playthrough asserts only that `scores` *is* a dict. +That thin prompt coverage is the standing weakness that makes an unscored +debrief reachable at all. It is *not*, however, the expected outcome: in local +verification against the real starter model — Qwen3 4B Q4\_K\_M, the same pin +the nightly uses, under the turn-output schema — every one of six scripted turns +across two runs volunteered at least one observation, and the debrief scored +three to four dimensions with an `overall_score` near 50. Treat zero +observations as a signal, not as the resting state. + So an unscored debrief means one of two quite different things: -- **`rubric_observations_seen` is 0 — the model left an unguided array empty.** - Nothing regressed; a 4 B Q4 model does this. Fix it in the product (give the - turn prompt a rubric layer, or play a scenario that defines one), not by - relaxing the assertion. +- **`rubric_observations_seen` is 0 — the model volunteered nothing.** Only the + prompt's bare schema hint was ever asking, so this is reachable by design — + but since the starter model does normally answer that hint, first check what + changed about what reaches the model: the `OUTPUT_SCHEMA` prompt layer, the + registry's `starter` pin, the adapter's JSON-schema constraint, sampling + settings. Then fix the weakness itself in the product (give the turn prompt a + rubric layer, or play a scenario that defines one), not by relaxing the + assertion. - **`rubric_observations_seen` is above 0 — a real regression.** The turns returned observations and the debrief scored none of them, so the model did its part. The debrief engine does not score the validated observations the @@ -139,9 +151,10 @@ So an unscored debrief means one of two quite different things: against the per-turn `rubric_observation_count` in the artifact. Either way the run is red: *"scored debrief"* is the acceptance criterion for -[#457](https://github.com/outrightmental/ConversationSimulator/issues/457), and -the first cause is a product gap worth a tracking issue rather than a reason to -weaken the check. +[#457](https://github.com/outrightmental/ConversationSimulator/issues/457). +Neither cause is a reason to weaken the check — the first is worth a tracking +issue for the missing rubric prompt layer *and* an investigation of the run that +hit it. --- diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index a7fa316a..8caa1ad3 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -169,25 +169,40 @@ class exit meaning MIN_SUMMARY_CHARS = 20 # Appended to the "no scores" failure. The debrief's dimension scores are -# accumulated purely from rubric_observations the *model* volunteers: the -# built-in behavioral_interview scenario defines no rubric, and no prompt layer -# names one (see packages/prompt-composer/.../layers.py — the model's only hint -# is the bare rubric_observations array in the embedded output schema, whose -# empty list the schema accepts). So an unscored debrief can mean the 4 B model -# simply left an unguided array empty rather than that anything regressed, and -# `pipeline`'s stock advice — inspect the per-turn used_fallback flags — would -# send triage looking for a parse failure that did not happen. +# accumulated purely from rubric_observations the *model* volunteers, and +# nothing asks it for them: the built-in behavioral_interview scenario defines +# no rubric, and no prompt layer names one (see +# packages/prompt-composer/.../layers.py — the model's only hint is the bare +# rubric_observations array in the embedded output schema, whose empty list the +# schema accepts). So `pipeline`'s stock advice — inspect the per-turn +# used_fallback flags — would send triage looking for a parse failure that did +# not happen. +# +# That unguided array is NOT normally empty, though, which is why this note +# does not simply write the failure off: in local verification against the real +# starter model (Qwen3 4B Q4_K_M under the turn-output schema) every one of six +# scripted turns volunteered at least one observation, and the debrief scored +# three to four dimensions. Zero observations across a whole conversation is +# therefore itself a change worth chasing — the schema layer no longer reaching +# the model, a sampling or quantisation change, a different starter pin — not +# the expected resting state. Say both halves: the thin prompt coverage is the +# standing weakness that makes this reachable at all, and a run that actually +# hits it is a signal. # # Used only when the run observed no rubric observations at all; when the turns # *did* carry some, the cause is not ambiguous and UNSCORED_WITH_OBSERVATIONS_NOTE # applies instead. UNSCORED_DEBRIEF_NOTE = ( - "Before calling this a regression: no NPC turn volunteered a single rubric " - "observation, and nothing asks it to — no prompt layer names the rubric " - "dimensions, so an empty array satisfies the turn schema. This is the " - "product gap, not a regression: fix it by giving the turn prompt a rubric " - "layer or by playing a scenario that defines one. See the 'Unscored " - "debrief' section of docs/real-model-smoke.md." + "No NPC turn volunteered a single rubric observation. Nothing asks it to — " + "no prompt layer names the rubric dimensions, so an empty array satisfies " + "the turn schema — which is the standing product weakness that makes this " + "failure reachable. But an empty array is not the normal outcome: the real " + "starter model volunteered observations on every scripted turn in local " + "verification. So check what changed about what reaches the model (the " + "OUTPUT_SCHEMA layer, the starter model pin, sampling) before writing this " + "off, and fix the weakness itself by giving the turn prompt a rubric layer " + "or playing a scenario that defines one. See the 'Unscored debrief' section " + "of docs/real-model-smoke.md." ) # The other cause of an unscored debrief, and the one that *is* a regression. @@ -780,7 +795,9 @@ def evaluate_debrief( actually returned, ``None`` when the run could not tell) resolves it: * **0** — the model volunteered nothing to score, which nothing asked it - to. The product gap, not a regression: ``UNSCORED_DEBRIEF_NOTE``. + to. Reachable because of the thin prompt coverage, but not the normal + outcome (the real starter model volunteers observations on every turn), + so still worth chasing: ``UNSCORED_DEBRIEF_NOTE``. * **> 0** — the turns carried observations and the debrief lost them. A real regression with a named starting point: ``UNSCORED_WITH_OBSERVATIONS_NOTE``. diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 28b7815d..9c8df0b4 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -397,7 +397,7 @@ def test_unscored_debrief_fails(self) -> None: assert any("no rubric dimension scores" in f for f in failures) assert any("not numeric" in f for f in failures) - def test_unscored_debrief_says_it_may_not_be_a_regression(self) -> None: + def test_unscored_debrief_carries_its_own_note_not_the_class_remedy(self) -> None: # The scores come only from rubric_observations the model volunteers, # and nothing in the prompt asks for them (the built-in scenario defines # no rubric and no prompt layer names one), so an empty array satisfies @@ -409,9 +409,14 @@ def test_unscored_debrief_says_it_may_not_be_a_regression(self) -> None: assert smoke.UNSCORED_DEBRIEF_NOTE in no_scores assert "real-model-smoke.md" in no_scores - def test_no_observations_anywhere_names_the_product_gap(self) -> None: + def test_no_observations_anywhere_is_named_and_still_worth_chasing(self) -> None: # The run checked, and the model volunteered nothing to score: the # failure can state that outright instead of listing both possibilities. + # It must not go on to excuse itself, though — the thin prompt coverage + # makes this reachable, but the real starter model answers the schema's + # bare hint on every turn, so a run that hits it changed something. A + # note that read "product gap, not a regression" would close the + # investigation the nightly exists to open. failures, _ = smoke.evaluate_debrief( _debrief(scores={}), rubric_observations_seen=0 ) @@ -419,6 +424,8 @@ def test_no_observations_anywhere_names_the_product_gap(self) -> None: assert "no NPC turn carried a rubric_observation" in no_scores assert smoke.UNSCORED_DEBRIEF_NOTE in no_scores assert smoke.UNSCORED_WITH_OBSERVATIONS_NOTE not in no_scores + assert "not the normal outcome" in no_scores + assert "before writing this off" in no_scores def test_observations_that_never_reached_the_debrief_are_a_regression(self) -> None: # The opposite case, and the one the harness exists to catch: the turns From ab0cb707f918b7f8af3693a28ea4e60d8d3e7e43 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 21:25:42 -0400 Subject: [PATCH 22/58] Refuse to run the smoke on a port it does not own Readiness is a URL poll, so a server the harness did not start answers it. `llama_cpp.server` loads the whole GGUF before it binds, so the child we spawned is still starting up when the first poll runs: the poll succeeds against the *foreign* server and `_wait_for_http` returns happy. Nothing downstream catches it either -- `_crashed_child` is only consulted once a SmokeFailure has been raised, so a run that otherwise goes green never notices that the process it started is dead. That produces the one verdict this job must never return: a pass earned by a model whose bytes it never checksum-verified, on a runtime it did not choose. The whole point of #457 is proving a real pinned model drove the conversation, and the harness is careful about this everywhere else -- it refuses a fake runtime, re-verifies the checksum on cache hits, and warns when the per-turn parse flags are missing so an absent proof is not read as a passed one. Port ownership was the remaining hole in the same chain of custody. Demonstrated before the fix: a child that sleeps 3 s and then fails to bind 7356 (occupied) leaves `_wait_for_http` reporting success while the child it started exits 1; `run_smoke` then returns exit 0 with a full scored report. After the fix the same setup is exit 4, class `runtime`, phase `runtime_start` -- verified by hand against a real llama-server listening on 7356, and covered by four tests. Both ports are checked before anything is spawned, and the failure carries its own remedy rather than `runtime`'s: the class advice is "read the child stderr tail", and here no child has been started. CI runners are clean, so this is about the local repro in docs/real-model-smoke.md -- which is also the only way the real-model path gets exercised before a change to it ships, since no PR job can run it. The doc said "stop anything already listening there first"; it now says the harness enforces that, and why not to work around it. The `fake_servers` fixture and the one orchestration test that builds its own children stub the check out: they spawn nothing, so leaving it live would make them pass or fail on whether the developer happens to have a llama-server on 7356 -- which, on the machine this was written on, they did. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 8 +- docs/real-model-smoke.md | 8 +- scripts/nightly-model-smoke.py | 41 +++++++ tests/scripts/test_nightly_model_smoke.py | 100 ++++++++++++++++++ 4 files changed, 153 insertions(+), 4 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 38b24aa0..90d23645 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -294,8 +294,12 @@ Useful extras: `~/.convsim/models/llm/`. - `--wall-clock-budget-s` — the self-imposed deadline (default 1200 s). -The harness binds `llama-server` on port 7356 and `convsim-core` on port 7399; -stop anything already listening there first. +The harness binds `llama-server` on port 7356 and `convsim-core` on port 7399, +and refuses to start (exit 4) if either is already taken. That is deliberate +rather than fussy: readiness is a URL poll, so a server the harness did not +start would answer it, the child that lost the bind would die unnoticed, and the +run would report a **pass** for a model it never checksum-verified. Stop whatever +owns the port — do not work around it. To exercise the harness's own logic without a model: diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 8b465687..ecdd37dc 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -290,8 +290,12 @@ Useful extras: `~/.convsim/models/llm/`. - `--wall-clock-budget-s` — the self-imposed deadline (default 1200 s). -The harness binds `llama-server` on port 7356 and `convsim-core` on port 7399; -stop anything already listening there first. +The harness binds `llama-server` on port 7356 and `convsim-core` on port 7399, +and refuses to start (exit 4) if either is already taken. That is deliberate +rather than fussy: readiness is a URL poll, so a server the harness did not +start would answer it, the child that lost the bind would die unnoticed, and the +run would report a **pass** for a model it never checksum-verified. Stop whatever +owns the port — do not work around it. To exercise the harness's own logic without a model: diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 8caa1ad3..23dd512f 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -100,6 +100,7 @@ class exit meaning import json import os import shutil +import socket import statistics import subprocess import sys @@ -558,6 +559,41 @@ def download_model(url: str, sha256: str, model_id: str, models_dir: Path = MODE # ── Server helpers ──────────────────────────────────────────────────────────── +def _assert_port_free(port: int, label: str) -> None: + """Refuse to run when something already listens on ``port``. + + Readiness is a URL poll, and a squatter answers it. The child we spawned + loses the bind and exits, but it is still starting up when the first poll + runs — ``llama_cpp.server`` loads the whole GGUF before it binds — so the + poll succeeds against the *foreign* server and ``_wait_for_http`` returns + happy. Nothing downstream catches it either: ``_crashed_child`` is only + consulted once a ``SmokeFailure`` has been raised, so a run that otherwise + goes green never notices that the process it started is dead. + + The result is the one verdict this job must never return: a pass earned by + a model whose bytes it never checksum-verified, on a runtime it did not + choose. CI runners are clean, so this is about the local repro in + docs/real-model-smoke.md — which is also how the real-model path gets + verified by hand, since no PR job can exercise it. + """ + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: + sock.settimeout(1.0) + if sock.connect_ex(("127.0.0.1", port)) != 0: + return + raise SmokeFailure( + FailureClass.RUNTIME, + f"Port {port} is already in use, so {label} cannot be started on it", + # Not the class remedy: it says to read the child's stderr tail, and no + # child has been started yet. + remedy=( + f"Stop whatever is listening on 127.0.0.1:{port}, then re-run. The smoke " + f"has to own {label}: readiness is a URL poll, so a server it did not " + "start answers it and the run would pass on a model these bytes were " + "never verified against." + ), + ) + + def _start_llama_server(model_path: Path, port: int) -> subprocess.Popen: """Start llama-cpp-python's OpenAI-compatible server on the given port. @@ -1156,6 +1192,11 @@ def run_smoke( # ── runtime ─────────────────────────────────────────────────────────── clock.enter("runtime_start") + # Both ports before anything is spawned: the smoke has to serve itself, + # or its proof that a real pinned model drove the conversation is worth + # nothing. + _assert_port_free(LLAMA_SERVER_PORT, "llama-server") + _assert_port_free(CORE_PORT, "convsim-core") print(f"\n[smoke] Starting llama-server on port {LLAMA_SERVER_PORT} " f"with model: {model_path.name}") llama_proc = _start_llama_server(model_path, LLAMA_SERVER_PORT) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 9c8df0b4..fbd22734 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -16,6 +16,7 @@ import importlib.util import io import json +import socket import statistics import urllib.error from pathlib import Path @@ -1069,6 +1070,97 @@ def test_an_http_error_still_means_the_server_is_up( ) is None +# --------------------------------------------------------------------------- +# Port ownership +# --------------------------------------------------------------------------- + + +@pytest.fixture() +def occupied_port(): + """A loopback port with a live listener on it, released after the test.""" + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.bind(("127.0.0.1", 0)) + sock.listen(1) + try: + yield sock.getsockname()[1] + finally: + sock.close() + + +@pytest.fixture() +def free_port() -> int: + """A loopback port nothing is listening on.""" + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: + sock.bind(("127.0.0.1", 0)) + return sock.getsockname()[1] + + +class TestPortOwnership: + """The smoke must own its ports, or its proof of a real model is worthless. + + Readiness is a URL poll, so a server the harness did not start answers it + and the run proceeds against a model whose bytes were never verified. The + child that lost the bind dies, but ``_crashed_child`` is only consulted + once a failure has been raised, so a run that otherwise passes never + notices -- it just reports a green verdict for the wrong model. + """ + + def test_a_free_port_is_accepted(self, free_port: int) -> None: + assert smoke._assert_port_free(free_port, "llama-server") is None + + def test_an_occupied_port_is_a_runtime_failure(self, occupied_port: int) -> None: + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._assert_port_free(occupied_port, "llama-server") + assert exc_info.value.failure_class == smoke.FailureClass.RUNTIME + assert exc_info.value.exit_code == 4 + assert str(occupied_port) in str(exc_info.value) + + def test_the_remedy_does_not_send_triage_to_an_unstarted_child( + self, occupied_port: int + ) -> None: + # `runtime`'s stock advice is "read the child stderr tail"; here no + # child has been started, so the class remedy would mislead. + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._assert_port_free(occupied_port, "convsim-core") + remedy = exc_info.value.remedy + assert "Stop whatever is listening" in remedy + assert remedy != smoke.REMEDIES[smoke.FailureClass.RUNTIME] + + def test_a_squatter_stops_the_run_before_any_server_is_started( + self, + staged_model, + occupied_port: int, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + ) -> None: + models_dir, model_id, digest = staged_model + started: list[str] = [] + + def _record(label: str): + def _start(*args: object, **kwargs: object) -> "_FakeProc": + started.append(label) + return _FakeProc() + return _start + + monkeypatch.setattr(smoke, "LLAMA_SERVER_PORT", occupied_port) + monkeypatch.setattr(smoke, "_start_llama_server", _record("llama")) + monkeypatch.setattr(smoke, "_start_core", _record("core")) + monkeypatch.setattr(smoke, "_wait_for_http", lambda *a, **k: None) + monkeypatch.setattr(smoke, "_request_json", _fake_core(_debrief())) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == 4 + assert started == [], "the squatter must be caught before anything is spawned" + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failure_class"] == smoke.FailureClass.RUNTIME + assert results["failed_phase"] == "runtime_start" + assert "Stop whatever is listening" in results["remedy"] + + # --------------------------------------------------------------------------- # Full-run orchestration, with llama-server and convsim-core faked out # --------------------------------------------------------------------------- @@ -1165,6 +1257,11 @@ def fake_servers(monkeypatch: pytest.MonkeyPatch) -> dict[str, _FakeProc]: monkeypatch.setattr(smoke, "_start_llama_server", lambda *a, **k: procs["llama"]) monkeypatch.setattr(smoke, "_start_core", lambda *a, **k: procs["core"]) monkeypatch.setattr(smoke, "_wait_for_http", lambda *a, **k: None) + # No child is really spawned, so no port is really needed. Left live, the + # port check would make every test below depend on whether the developer + # happens to have a llama-server on 7356 -- which, on the machine this was + # written on, they did. TestPortOwnership covers the check itself. + monkeypatch.setattr(smoke, "_assert_port_free", lambda *a, **k: None) return procs @@ -1239,6 +1336,9 @@ def _start_core(data_dir, *args, **kwargs) -> _FakeProc: monkeypatch.setattr(smoke, "_start_llama_server", lambda *a, **k: _FakeProc()) monkeypatch.setattr(smoke, "_start_core", _start_core) monkeypatch.setattr(smoke, "_wait_for_http", lambda *a, **k: None) + # Builds its own children rather than using fake_servers, so it has to + # neutralise the port check the same way that fixture does. + monkeypatch.setattr(smoke, "_assert_port_free", lambda *a, **k: None) exit_code = smoke.run_smoke( model_id, 20.0, None, model_sha256=digest, models_dir=models_dir From c59fae197d81e2a7d65fb40e67aa44fa108b3abe Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 21:30:12 -0400 Subject: [PATCH 23/58] Stop the nightly's log from printing the verdict before the evidence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI captures this harness's stdout through a pipe, so Python block-buffers it (8 KB) while stderr stays unbuffered. Everything the run narrates — the phase it is in, each turn's latency, the budget arithmetic — goes to stdout; the failure banner, its remedy and the child stderr tails go to stderr. Merged into one job log, the result reads backwards: the verdict lands first and the context that explains it arrives later, or only when the process exits. Reproduced with the harness itself. Piped, a --verify-only drift prints the CHECKSUM banner on line 2 and "Verifying cached model drifted…" on the last line, 900 characters further down. The same buffering also means nothing streams. A run spends ~11 min inside the conversation and debrief phases, so a nightly that hangs would show a log that stops at "Starting llama-server" and says nothing about which of the three scripted turns it reached — exactly the attribution the wall-clock deadline and the per-phase durations exist to provide. Reconfigure both streams to line buffering at the top of main(), so the local repro in docs/real-model-smoke.md behaves the same way as CI, and tolerate a stdout that cannot be reconfigured (pytest's capture, contextlib.redirect_stdout) rather than failing on it. Covered by TestLogOrdering, which runs the script as a subprocess with stdout and stderr merged into one pipe — the only way to observe the interleaving the job log actually shows. It fails without the fix. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 25 +++++++++ tests/scripts/test_nightly_model_smoke.py | 63 +++++++++++++++++++++++ 2 files changed, 88 insertions(+) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 23dd512f..f703921b 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1548,7 +1548,32 @@ def error(self, message: str) -> NoReturn: sys.exit(EXIT_CODES[FailureClass.PIPELINE]) +def _use_line_buffered_output() -> None: + """Make stdout arrive in step with stderr, line by line. + + CI captures this script's stdout through a pipe, so Python block-buffers it + (8 KB) while stderr stays unbuffered. Two things break as a result, and + both are the log a triager actually reads: + + * nothing streams. A run spends ~11 min inside the conversation and + debrief phases, and the "Player turn 2/3…" lines that say where it got to + only reach the log when the buffer fills or the process exits. + * the failure banner arrives *before* the context it explains. The banner + and the child stderr tails go to stderr; the phase progress, per-turn + latencies and budget arithmetic go to stdout, so the merged log shows the + verdict first and the evidence afterwards. + """ + for stream in (sys.stdout, sys.stderr): + try: + stream.reconfigure(line_buffering=True) # type: ignore[union-attr] + except (AttributeError, ValueError): + # Not a TextIOWrapper (pytest's capture, a redirect_stdout buffer). + # Buffering is then the caller's concern, not ours. + pass + + def main(argv: Optional[Sequence[str]] = None) -> int: + _use_line_buffered_output() parser = _ClassifiedArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter ) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index fbd22734..b4db8f07 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -2008,3 +2008,66 @@ def test_a_usage_error_is_not_mistaken_for_a_download_failure(self) -> None: smoke.main(["--download-only", "--model-id", "x"]) # no url / sha256 assert exc_info.value.code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] assert exc_info.value.code != smoke.EXIT_CODES[smoke.FailureClass.DOWNLOAD] + + +# --------------------------------------------------------------------------- +# Log ordering +# --------------------------------------------------------------------------- + + +class TestLogOrdering: + """The CI log has to read in the order things happened. + + CI captures the harness through a pipe, so Python block-buffers stdout + while stderr stays unbuffered. Unfixed, that means the phase progress and + per-turn latencies (stdout) arrive after the failure banner and child + stderr tails (stderr) — and during the ~11 min the conversation and + debrief phases take, nothing streams at all. + """ + + def test_progress_is_logged_before_the_banner_it_explains( + self, tmp_path: Path + ) -> None: + import subprocess + import sys + + model = tmp_path / "drifted.gguf" + model.write_bytes(b"not the pinned bytes") + + # Piped, merged, and run with the inherited environment so that only + # the harness's own buffering decides the order. + proc = subprocess.run( + [ + # The interpreter running the tests, not whatever "python3" + # resolves to on PATH: the harness imports PyYAML, which is + # only guaranteed in this environment. + sys.executable, str(SCRIPT_PATH), "--verify-only", + "--model-id", "drifted", "--model-sha256", "0" * 64, + "--models-dir", str(tmp_path), + ], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + timeout=120, + ) + + assert proc.returncode == smoke.EXIT_CODES[smoke.FailureClass.CHECKSUM] + log = proc.stdout + progress_at = log.find("Verifying cached model") + banner_at = log.find("FAILED — class: CHECKSUM") + assert progress_at != -1, f"progress line missing from:\n{log}" + assert banner_at != -1, f"banner missing from:\n{log}" + assert progress_at < banner_at, ( + "the stdout progress line arrived after the stderr banner, so the " + f"CI log reads out of order:\n{log}" + ) + + def test_reconfiguring_a_stream_that_cannot_be_reconfigured_is_survivable( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + # Under pytest's capture, and under contextlib.redirect_stdout, stdout + # is not a TextIOWrapper. Buffering is then the caller's business — but + # it must not take the harness down on the way past. + monkeypatch.setattr("sys.stdout", io.StringIO()) + monkeypatch.setattr("sys.stderr", io.StringIO()) + smoke._use_line_buffered_output() From a97465481ad5898adc03f19e5e89b41349dd68ec Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 21:30:58 -0400 Subject: [PATCH 24/58] Say in the failure table that exit 4 also means a port was taken MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doc bills the failure table as the triage entry point, and its `runtime` row sends the reader to "the child stderr tail printed above the banner". That is the right first move for the three causes the row lists, but not for the fourth: a port conflict is refused before either server is spawned, so there is no child and no tail, and a reader following the table looks for output that does not exist. The harness itself already gets this right — the port failure carries its own remedy, naming the port to free, instead of the class default. The table was the only place still describing exit 4 as nothing but a crash. Co-Authored-By: Claude Opus 5 --- docs-site/src/content/docs/dev/real-model-smoke.md | 2 +- docs/real-model-smoke.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 90d23645..a8528e47 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -70,7 +70,7 @@ log, and a remedy printed next to it: | 1 | `budget` | Pipeline worked; latency regressed past the CI ceiling | Compare `measured_ms` in the report artifact against recent nightlies | | 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | -| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx | Read the child stderr tail printed above the banner | +| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner. A port conflict started no child and so has no tail: the banner carries its own remedy instead, naming the port to free — see [Running it locally](#running-it-locally) | | 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | | 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index ecdd37dc..d552a280 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -66,7 +66,7 @@ log, and a remedy printed next to it: | 1 | `budget` | Pipeline worked; latency regressed past the CI ceiling | Compare `measured_ms` in the report artifact against recent nightlies | | 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | -| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx | Read the child stderr tail printed above the banner | +| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner. A port conflict started no child and so has no tail: the banner carries its own remedy instead, naming the port to free — see [Running it locally](#running-it-locally) | | 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | | 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | From 91baa1f6472720cd9dbcaa58021217e68f3bd02b Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 21:38:15 -0400 Subject: [PATCH 25/58] Stop the unscored-debrief verdict from asserting the cause it cannot know MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit evaluate_debrief() resolves an unscored debrief into one of two causes from rubric_observations_seen: the model volunteered nothing to score, or the debrief engine dropped observations the turns did return. When the count is unavailable it cannot know which, and said so — then appended a note opening "No NPC turn volunteered a single rubric observation", contradicting the sentence before it and sending triage after the product weakness while the regression half of the fork went unmentioned. The one piece of triage this harness exists to have already done, done wrongly. The note no longer claims the array was empty; the caller's prefix states what the run actually observed, and the unknown branch now quotes both causes and points at the per-turn rubric_observation_count in the report artifact that tells them apart. The 0 and >0 branches read as before. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 24 +++++++++++++++++------ tests/scripts/test_nightly_model_smoke.py | 13 ++++++++++++ 2 files changed, 31 insertions(+), 6 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index f703921b..bc08477d 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -190,11 +190,15 @@ class exit meaning # standing weakness that makes this reachable at all, and a run that actually # hits it is a signal. # -# Used only when the run observed no rubric observations at all; when the turns -# *did* carry some, the cause is not ambiguous and UNSCORED_WITH_OBSERVATIONS_NOTE -# applies instead. +# Used when the run observed no rubric observations at all, and again when it +# could not tell; when the turns *did* carry some, the cause is not ambiguous +# and UNSCORED_WITH_OBSERVATIONS_NOTE applies instead. The note therefore +# explains the weakness without itself asserting that the array was empty — +# the caller's own prefix says what the run actually observed, and a note that +# announced "No NPC turn volunteered a single rubric observation" contradicted +# the prefix on the branch that had just said it could not tell. UNSCORED_DEBRIEF_NOTE = ( - "No NPC turn volunteered a single rubric observation. Nothing asks it to — " + "Nothing asks the NPC for rubric observations — " "no prompt layer names the rubric dimensions, so an empty array satisfies " "the turn schema — which is the standing product weakness that makes this " "failure reachable. But an empty array is not the normal outcome: the real " @@ -860,9 +864,17 @@ def evaluate_debrief( "had nothing to accumulate. " ) + UNSCORED_DEBRIEF_NOTE else: + # Neither cause can be ruled out, so quote both rather than pick + # one: asserting the model volunteered nothing, on the one branch + # that just said it could not tell, is the triage this harness is + # supposed to have done, done wrongly. cause = ( - "the run could not tell whether any NPC turn carried a " - "rubric_observation. " + UNSCORED_DEBRIEF_NOTE + "the run did not record how many rubric observations the NPC " + "turns carried, so the two causes below cannot be told apart " + "from this message — read rubric_observation_count per turn in " + "the report artifact. " + UNSCORED_DEBRIEF_NOTE + + " If the turns did carry observations, it is the other cause " + "instead: " + UNSCORED_WITH_OBSERVATIONS_NOTE ) failures.append(f"Debrief has no rubric dimension scores: {cause}") diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index b4db8f07..282955a3 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -428,6 +428,19 @@ def test_no_observations_anywhere_is_named_and_still_worth_chasing(self) -> None assert "not the normal outcome" in no_scores assert "before writing this off" in no_scores + def test_an_unknown_observation_count_claims_neither_cause(self) -> None: + # The count is what tells the two causes apart, so without it the + # failure has to quote both. It must not state one as fact: a message + # that said "could not tell" and then "No NPC turn volunteered a single + # rubric observation" sends triage after the product weakness when the + # debrief engine dropping validated observations is just as open. + failures, _ = smoke.evaluate_debrief(_debrief(scores={})) + no_scores = next(f for f in failures if "no rubric dimension scores" in f) + assert "cannot be told apart" in no_scores + assert smoke.UNSCORED_DEBRIEF_NOTE in no_scores + assert smoke.UNSCORED_WITH_OBSERVATIONS_NOTE in no_scores + assert "No NPC turn volunteered" not in no_scores + def test_observations_that_never_reached_the_debrief_are_a_regression(self) -> None: # The opposite case, and the one the harness exists to catch: the turns # returned observations and the debrief scored nothing, so the model did From 6ba9a6938956a2eb478271b179d11f861a7062c0 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 23:04:51 -0400 Subject: [PATCH 26/58] Pin the two checksum promises that live in the workflow, not the harness "Fail loudly on checksum drift" (#457) is only half enforced by scripts/nightly-model-smoke.py. The other half is step wiring that no test touched, so either could be dropped in a one-line YAML edit while the whole suite stayed green: - The full smoke re-verifies the file it is about to load only when the step passes --model-sha256. Without the flag the harness downgrades to a warning and loads whatever the cache restored -- which is exactly the drifted cache hit this job exists to catch, now passing instead of exiting 3. - The harness cannot keep a rejected GGUF out of the Actions cache; only the split restore/save gated on success() can. actions/cache's post step saves even on a failed job, and the key is derived from the expected hash, so one poisoned entry would be restored by every later run. Adds four guards over the parsed workflow: the smoke step passes --model-sha256; that hash and the cache key resolve to the same registry lookup (two lookups can drift apart and cache a file under a hash nothing checked); the combined actions/cache action is not used, exactly one save step exists, it is gated on success() and ordered after the verifying download; and restore and save agree on key and path, so a save can actually produce a hit rather than silently re-downloading 2.5 GB every night. Each guard was confirmed to fail against a workflow with the invariant removed. Co-Authored-By: Claude Opus 5 --- tests/scripts/test_nightly_model_smoke.py | 87 +++++++++++++++++++++++ 1 file changed, 87 insertions(+) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 282955a3..45ea3ec9 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -675,6 +675,93 @@ def test_the_default_matches_what_the_workflow_passes(self) -> None: assert _workflow_budget_s() == smoke.DEFAULT_WALL_CLOCK_BUDGET_S +def _workflow_steps() -> list: + import yaml + + return yaml.safe_load(_workflow_text())["jobs"]["model-smoke"]["steps"] + + +def _step_running(fragment: str) -> dict: + """Return the single workflow step whose ``run`` contains ``fragment``.""" + matches = [s for s in _workflow_steps() if fragment in (s.get("run") or "")] + assert len(matches) == 1, f"expected exactly one step running {fragment!r}" + return matches[0] + + +def _steps_using(prefix: str) -> list: + return [s for s in _workflow_steps() if (s.get("uses") or "").startswith(prefix)] + + +def _step_index(step: dict) -> int: + return _workflow_steps().index(step) + + +class TestWorkflowChecksumInvariants: + """#457's "fail loudly on checksum drift" is enforced by the YAML, not the harness. + + Two halves of that promise live in the workflow and nowhere else, and each + is one quiet edit away from being given up while every other test in this + file stays green: + + * ``verify_model_checksum`` only runs on a full smoke when + ``--model-sha256`` is passed. Drop the flag and the harness downgrades to + a warning and loads whatever the cache restored — the exact cache-hit + drift this job exists to catch. + * The harness cannot keep a GGUF it rejected out of the Actions cache; only + the step ordering can. ``actions/cache``'s post step saves even when the + job failed, and the key is derived from the *expected* hash, so a single + poisoned entry would be restored by every later run. + """ + + def test_the_smoke_reverifies_the_file_it_is_about_to_load(self) -> None: + run = _step_running("--ci-hardware-factor")["run"] + assert "--model-sha256" in run, ( + "without --model-sha256 the full smoke only warns that drift was " + "not checked, so a corrupt cache hit is fed to the model and the " + "run can still go green" + ) + + def test_the_cache_key_and_the_verified_hash_come_from_one_lookup(self) -> None: + import re + + pattern = r"steps\.([A-Za-z0-9_-]+)\.outputs\.model_sha256" + restore = _steps_using("actions/cache/restore") + assert restore, "the model cache must be restored" + key_sources = {m for s in restore for m in re.findall(pattern, s["with"]["key"])} + smoke_sources = set(re.findall(pattern, _step_running("--ci-hardware-factor")["run"])) + assert key_sources and key_sources == smoke_sources, ( + "the cache key and the hash the smoke verifies against must come " + f"from the same registry lookup (key: {key_sources}, smoke: {smoke_sources}); " + "two lookups can drift apart and cache a file under a hash nothing checked" + ) + + def test_an_unverified_model_is_never_written_to_the_cache(self) -> None: + assert not _steps_using("actions/cache@"), ( + "actions/cache saves from a post step that runs even when the job " + "failed; use the split restore/save so the save can be gated" + ) + save = _steps_using("actions/cache/save") + assert len(save) == 1, "exactly one step may write the model cache" + assert "success()" in str(save[0].get("if", "")), ( + "the cache save must be gated on success(), or a GGUF that failed " + "checksum verification is cached under its expected hash and every " + "later run restores the same bad bytes" + ) + download = _step_running("--download-only") + assert _step_index(download) < _step_index(save[0]), ( + "the model must be downloaded and verified before it is cached" + ) + + def test_the_restored_and_saved_cache_entries_are_the_same(self) -> None: + restore = _steps_using("actions/cache/restore")[0] + save = _steps_using("actions/cache/save")[0] + assert restore["with"]["key"] == save["with"]["key"], ( + "a save under a different key than the restore never produces a " + "cache hit, so every nightly re-downloads 2.5 GB" + ) + assert restore["with"]["path"] == save["with"]["path"] + + class TestDeadline: def test_phase_durations_are_recorded(self) -> None: clock = smoke.Deadline(60.0) From dae25f936d82e7daa80a1b03e6653c89ddd074b1 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 23:08:05 -0400 Subject: [PATCH 27/58] Stop an unreadable turn payload from reading as "the model scored nothing" The unscored-debrief verdict picks between two opposite remedies using the rubric observation counts the NPC turns returned, so that count has to mean what the verdict says it means. _rubric_observation_count returned 0 both for an empty rubric_observations list -- the model was asked by the schema's bare hint and volunteered nothing -- and for a payload with no readable list at all, which is a turn response contract that changed under us. The second case got the first case's verdict: "no NPC turn carried a rubric_observation", sending triage after the OUTPUT_SCHEMA prompt layer, the starter model pin and sampling settings for a cause the run had no evidence for. That is the same mistake the per-turn parse flags already avoid -- absent flags warn that the fallback check did not run rather than passing it -- and evaluate_debrief already had the branch for it: a None count quotes both causes and sends the reader to the per-turn numbers. Nothing could ever reach it, because run_smoke summed the counts with a 0 default. So an unreadable list now yields None, and _total_rubric_observations folds the per-turn counts into None only when nothing readable was found. A positive total still settles the diagnosis even with a gap among the turns: observations were seen, so the debrief losing them is the cause whatever the missing turn carried. The authored opening is skipped, and a run with no generated turn at all reports None rather than claiming it observed zero. Also makes the baseline fake turn in the tests carry the empty rubric_observations list that convsim-core actually sends, so the fixture matches the contract rather than exercising the unreadable path by accident. Docs: the "Unscored debrief" section lists the third case and why an unreadable payload is not scored as zero. --- .../src/content/docs/dev/real-model-smoke.md | 17 +++- docs/real-model-smoke.md | 17 +++- scripts/nightly-model-smoke.py | 55 +++++++++-- tests/scripts/test_nightly_model_smoke.py | 94 +++++++++++++++++-- 4 files changed, 157 insertions(+), 26 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index a8528e47..c179411a 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -116,7 +116,7 @@ Exit 5 with *"Debrief has no rubric dimension scores"* is the one `pipeline` failure whose cause is ambiguous from the class alone, so the harness resolves it for you: each NPC turn's `rubric_observation_count` is recorded in the report artifact, totalled as `rubric_observations_seen`, and the failure message names -whichever of the two causes below applies. +which of the causes below applies. The debrief's dimension scores are accumulated entirely from `rubric_observations` that the model volunteers on each NPC turn. Nothing asks @@ -134,7 +134,7 @@ across two runs volunteered at least one observation, and the debrief scored three to four dimensions with an `overall_score` near 50. Treat zero observations as a signal, not as the resting state. -So an unscored debrief means one of two quite different things: +So an unscored debrief means one of three quite different things: - **`rubric_observations_seen` is 0 — the model volunteered nothing.** Only the prompt's bare schema hint was ever asking, so this is reachable by design — @@ -153,10 +153,17 @@ So an unscored debrief means one of two quite different things: that arrived pre-parsed from the runtime adapter — loses its observations on the way to the debrief. Compare `raw_npc_output` from the `/debug` payload against the per-turn `rubric_observation_count` in the artifact. - -Either way the run is red: *"scored debrief"* is the acceptance criterion for +- **`rubric_observations_seen` is `null` — the harness could not tell.** The NPC + turn payloads carried no readable `rubric_observations` list, so the per-turn + `rubric_observation_count` is `null` too and the message quotes both causes + above rather than picking one. `convsim-core` always sends that list today, so + this is itself worth chasing: the turn response contract changed. The harness + deliberately does not score an unreadable payload as zero — that would have + the verdict name the one cause it has no evidence for. + +In every case the run is red: *"scored debrief"* is the acceptance criterion for [#457](https://github.com/outrightmental/ConversationSimulator/issues/457). -Neither cause is a reason to weaken the check — the first is worth a tracking +No cause is a reason to weaken the check — the first is worth a tracking issue for the missing rubric prompt layer *and* an investigation of the run that hit it. diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index d552a280..12040889 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -112,7 +112,7 @@ Exit 5 with *"Debrief has no rubric dimension scores"* is the one `pipeline` failure whose cause is ambiguous from the class alone, so the harness resolves it for you: each NPC turn's `rubric_observation_count` is recorded in the report artifact, totalled as `rubric_observations_seen`, and the failure message names -whichever of the two causes below applies. +which of the causes below applies. The debrief's dimension scores are accumulated entirely from `rubric_observations` that the model volunteers on each NPC turn. Nothing asks @@ -130,7 +130,7 @@ across two runs volunteered at least one observation, and the debrief scored three to four dimensions with an `overall_score` near 50. Treat zero observations as a signal, not as the resting state. -So an unscored debrief means one of two quite different things: +So an unscored debrief means one of three quite different things: - **`rubric_observations_seen` is 0 — the model volunteered nothing.** Only the prompt's bare schema hint was ever asking, so this is reachable by design — @@ -149,10 +149,17 @@ So an unscored debrief means one of two quite different things: that arrived pre-parsed from the runtime adapter — loses its observations on the way to the debrief. Compare `raw_npc_output` from the `/debug` payload against the per-turn `rubric_observation_count` in the artifact. - -Either way the run is red: *"scored debrief"* is the acceptance criterion for +- **`rubric_observations_seen` is `null` — the harness could not tell.** The NPC + turn payloads carried no readable `rubric_observations` list, so the per-turn + `rubric_observation_count` is `null` too and the message quotes both causes + above rather than picking one. `convsim-core` always sends that list today, so + this is itself worth chasing: the turn response contract changed. The harness + deliberately does not score an unreadable payload as zero — that would have + the verdict name the one cause it has no evidence for. + +In every case the run is red: *"scored debrief"* is the acceptance criterion for [#457](https://github.com/outrightmental/ConversationSimulator/issues/457). -Neither cause is a reason to weaken the check — the first is worth a tracking +No cause is a reason to weaken the check — the first is worth a tracking issue for the missing rubric prompt layer *and* an investigation of the run that hit it. diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index bc08477d..506d22f3 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -741,7 +741,7 @@ def _npc_turn_content(events: Sequence[dict]) -> str: return content if isinstance(content, str) else "" -def _rubric_observation_count(events: Sequence[dict]) -> int: +def _rubric_observation_count(events: Sequence[dict]) -> Optional[int]: """Count the rubric observations the model volunteered on this NPC turn. The turn response carries the *validated* observations, which is the only @@ -751,9 +751,37 @@ def _rubric_observation_count(events: Sequence[dict]) -> int: were accepted on the turn but never scored mean the debrief engine lost them, while no observations anywhere means the model simply never volunteered any. See ``evaluate_debrief``. + + Returns ``None`` — not 0 — when the payload carries no readable + ``rubric_observations`` list, the same way the per-turn parse flags warn + rather than pass when the debug endpoint gives us nothing: "the model + volunteered none" and "we could not read what it volunteered" lead to + opposite triage, and the whole point of recording this is that the + unscored-debrief verdict names a cause it has actually established. An + empty list is a real observation of zero and still counts as 0. """ observations = _npc_turn_payload(events).get("rubric_observations") - return len(observations) if isinstance(observations, list) else 0 + return len(observations) if isinstance(observations, list) else None + + +def _total_rubric_observations(turns: Sequence[Dict[str, Any]]) -> Optional[int]: + """Total the per-turn rubric observation counts, or ``None`` if unknowable. + + ``None`` only when *nothing* readable was found — an unreadable payload, or + no generated turn to read at all. A total above zero already settles which + cause of an unscored debrief applies (the turns did carry observations), so + an unreadable turn alongside readable ones does not make the verdict + ambiguous. Only the authored opening is skipped: it is not a generation and + so has no count to miss. + """ + counts = [ + t.get("rubric_observation_count") for t in turns if t.get("model_generated") + ] + total = sum(c for c in counts if isinstance(c, int)) + if total > 0: + return total + nothing_readable = not counts or any(not isinstance(c, int) for c in counts) + return None if nothing_readable else 0 def _excerpt(text: str) -> str: @@ -832,7 +860,8 @@ def evaluate_debrief( failure — but on its own that failure does not say which of two very different things happened, and the remedy differs completely. ``rubric_observations_seen`` (how many rubric observations the NPC turns - actually returned, ``None`` when the run could not tell) resolves it: + actually returned, ``None`` when the run could not read any — see + ``_total_rubric_observations``) resolves it: * **0** — the model volunteered nothing to score, which nothing asked it to. Reachable because of the thin prompt coverage, but not the normal @@ -869,10 +898,12 @@ def evaluate_debrief( # that just said it could not tell, is the triage this harness is # supposed to have done, done wrongly. cause = ( - "the run did not record how many rubric observations the NPC " + "the run could not read how many rubric observations the NPC " "turns carried, so the two causes below cannot be told apart " "from this message — read rubric_observation_count per turn in " - "the report artifact. " + UNSCORED_DEBRIEF_NOTE + "the report artifact (null means the turn payload carried no " + "readable rubric_observations list, which is itself worth a " + "look at the turn response contract). " + UNSCORED_DEBRIEF_NOTE + " If the turns did carry observations, it is the other cause " "instead: " + UNSCORED_WITH_OBSERVATIONS_NOTE ) @@ -1389,15 +1420,19 @@ def run_smoke( "latency_ms": round(debrief_ms), } # Everything the NPC turns offered the debrief to score. An unscored - # debrief means something different depending on whether this is zero. - observations_seen = sum( - t.get("rubric_observation_count", 0) for t in results["turns"] - ) + # debrief means something different depending on whether this is zero — + # and something different again when we could not read it, so an + # unreadable turn payload yields None rather than a confident zero. + # A positive total is still conclusive even with an unreadable turn + # among the others: observations *were* seen, so the debrief losing + # them is the cause whatever the missing turn carried. + observations_seen = _total_rubric_observations(results["turns"]) results["rubric_observations_seen"] = observations_seen print(f"[smoke] Debrief in {debrief_ms:.0f} ms | " f"overall_score={debrief.get('overall_score')} " f"scores={debrief.get('scores')} " - f"rubric_observations_from_turns={observations_seen}") + f"rubric_observations_from_turns=" + f"{'unknown' if observations_seen is None else observations_seen}") # ── assertions ──────────────────────────────────────────────────────── clock.enter("assertions") diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 45ea3ec9..c5c09752 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -915,15 +915,50 @@ def test_rubric_observations_are_counted(self) -> None: }}] assert smoke._rubric_observation_count(events) == 2 + def test_an_empty_observation_list_counts_zero(self) -> None: + # An empty list is a real observation of zero: the model was asked (by + # the schema's bare hint) and volunteered nothing. + assert smoke._rubric_observation_count( + [{"event_type": "npc_turn", "payload": { + "content": "x", "rubric_observations": []}}] + ) == 0 + @pytest.mark.parametrize("payload", [ - {"content": "x"}, # key absent entirely - {"content": "x", "rubric_observations": []}, - {"content": "x", "rubric_observations": None}, + {"content": "x"}, # key absent entirely + {"content": "x", "rubric_observations": None}, # present but unusable + {"content": "x", "rubric_observations": "two"}, ]) - def test_absent_or_empty_rubric_observations_count_zero(self, payload: dict) -> None: + def test_an_unreadable_observation_list_is_unknown_not_zero( + self, payload: dict + ) -> None: + # Counting an unreadable payload as zero would let the unscored-debrief + # verdict assert "no NPC turn carried a rubric_observation" — sending + # triage after the prompt layer and the model pin — when what actually + # happened is that the turn response no longer exposes the list. assert smoke._rubric_observation_count( [{"event_type": "npc_turn", "payload": payload}] - ) == 0 + ) is None + + @pytest.mark.parametrize(("counts", "expected"), [ + ([0, 0, 0], 0), + ([1, 0, 2], 3), + ([None, None, None], None), # nothing readable anywhere + ([None, 0, 0], None), # a gap with no positive evidence + ([None, 2, 0], 2), # observations were seen regardless + ]) + def test_the_total_is_unknown_only_when_nothing_was_readable( + self, counts: list, expected: int | None + ) -> None: + turns = [{"model_generated": False, "label": "npc_opening"}] + [ + {"model_generated": True, "rubric_observation_count": c} for c in counts + ] + assert smoke._total_rubric_observations(turns) == expected + + def test_a_run_with_no_generated_turns_reports_an_unknown_total(self) -> None: + # Nothing was observed, which is not the same as observing nothing. + assert smoke._total_rubric_observations( + [{"model_generated": False, "label": "npc_opening"}] + ) is None def test_excerpt_is_bounded_and_collapsed(self) -> None: excerpt = smoke._excerpt("a\n\n b" + "x" * 500) @@ -1294,8 +1329,14 @@ def kill(self) -> None: # pragma: no cover - only on a hung child _OPENING = {"events": [{"event_type": "npc_opening", "payload": {"content": "Thanks for coming in today."}}]} +# convsim-core always puts a rubric_observations list in the npc_turn payload +# (see the npc_event payload in routers/sessions.py), empty or not, so the +# baseline fake carries one too. A payload *without* the key is a different +# situation the harness must not read as "the model volunteered none" — see +# test_a_turn_payload_without_observations_is_not_read_as_zero. _NPC_TURN = {"events": [{"event_type": "npc_turn", - "payload": {"content": "Walk me through that trade-off."}}], + "payload": {"content": "Walk me through that trade-off.", + "rubric_observations": []}}], "ending_type": None} @@ -1508,6 +1549,47 @@ def test_unscored_debrief_with_scorable_turns_is_named_a_regression( assert any(smoke.UNSCORED_WITH_OBSERVATIONS_NOTE in f for f in results["failures"]) assert not any(smoke.UNSCORED_DEBRIEF_NOTE in f for f in results["failures"]) + def test_a_turn_payload_without_observations_is_not_read_as_zero( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + """A third case: the harness could not read what the turns volunteered. + + convsim-core always sends a rubric_observations list today, so a payload + without one means the turn response contract changed. Scoring that as + "the model volunteered nothing" would have the verdict confidently name + the one cause the run has no evidence for, and send triage to the prompt + layer and the model pin. Say both causes are open instead. + """ + models_dir, model_id, digest = staged_model + no_observations_key = { + "events": [{"event_type": "npc_turn", + "payload": {"content": "Walk me through that trade-off."}}], + "ending_type": None, + } + monkeypatch.setattr( + smoke, "_request_json", + _fake_core( + _debrief(scores={}, overall_score=None), + turn=no_observations_key, + ), + ) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["rubric_observations_seen"] is None + assert all( + t["rubric_observation_count"] is None for t in results["turns"][1:] + ), "a null per-turn count is what tells the reader the list was unreadable" + no_scores = next(f for f in results["failures"] if "rubric dimension" in f) + assert "could not read" in no_scores + assert smoke.UNSCORED_DEBRIEF_NOTE in no_scores + assert smoke.UNSCORED_WITH_OBSERVATIONS_NOTE in no_scores + def test_latency_regression_is_a_budget_failure_not_a_pipeline_one( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: From a8cd38c91e7f3fb3c0949ba6fb68bc8e3c491f04 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 23:09:03 -0400 Subject: [PATCH 28/58] Say what a green nightly does not prove when the debrief narrative falls back The job exercises exactly two model-driven surfaces: the NPC turns and the debrief narrative. For the turns the harness has a rule -- every generated turn falling back is a failure, because then the smoke proved nothing. For the debrief it had a four-word warning, "Debrief narrative used the deterministic fallback, not model prose", and docs that said the job proves a scored debrief is produced. That reads as proof the model wrote it. It is not: the dimension scores are accumulated from the rubric observations the turns volunteered, not from the debrief generation, so a debrief whose prose came entirely from the template satisfies every assertion here. The debrief is generated once, so a fallback means no model-written debrief prose was produced at all -- and since the debrief prompt is the largest schema-constrained generation in the product, and no other job runs it against a real model, a regression there could stay green indefinitely behind a warning nobody reads on a passing run. Keeping it a warning rather than a failure: the scores are unaffected, and reddening the nightly on one unlucky generation would make it flap on the strength of two local runs' worth of evidence about the 4B model's fallback rate. What changes is that the warning and the docs now say what the verdict does not cover, and that a fallback recurring across nightlies is a regression in the debrief prompt or DEBRIEF_NARRATIVE_SCHEMA rather than noise -- so the limitation is legible from the step summary instead of from this file. --- .../src/content/docs/dev/real-model-smoke.md | 7 ++++++- docs/real-model-smoke.md | 7 ++++++- scripts/nightly-model-smoke.py | 20 ++++++++++++++++--- tests/scripts/test_nightly_model_smoke.py | 9 ++++++++- 4 files changed, 37 insertions(+), 6 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index c179411a..6a238aa3 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -44,7 +44,12 @@ on a **real local model** end-to-end — registry download → llama.cpp → `turn-output.schema.json`. 4. The session ends and produces a **scored** debrief: at least one rubric dimension scored, a numeric `overall_score` in `[0, 100]`, and a non-empty - summary. + summary. The scores come from the rubric observations the model volunteered + on each turn, so this asserts the scoring path; the debrief *narrative* is a + separate model call, and a run where it fell back to the deterministic + template still passes — with a warning that says so, because the scores are + unaffected and one unlucky generation must not red the nightly. Recurring + across nightlies is a regression in the debrief prompt or its schema. 5. End-to-end turn latency stays within the documented budget, scaled for CI hardware. diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 12040889..bd85e5b2 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -40,7 +40,12 @@ on a **real local model** end-to-end — registry download → llama.cpp → `turn-output.schema.json`. 4. The session ends and produces a **scored** debrief: at least one rubric dimension scored, a numeric `overall_score` in `[0, 100]`, and a non-empty - summary. + summary. The scores come from the rubric observations the model volunteered + on each turn, so this asserts the scoring path; the debrief *narrative* is a + separate model call, and a run where it fell back to the deterministic + template still passes — with a warning that says so, because the scores are + unaffected and one unlucky generation must not red the nightly. Recurring + across nightlies is a regression in the debrief prompt or its schema. 5. End-to-end turn latency stays within the documented budget, scaled for CI hardware. diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 506d22f3..f351344f 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -923,9 +923,23 @@ def evaluate_debrief( ) if debrief.get("used_fallback"): - # The scores are still real (they come from per-turn rubric observations), - # so this degrades the debrief narrative rather than invalidating the run. - warnings.append("Debrief narrative used the deterministic fallback, not model prose") + # Not a failure: the scores — the thing #457 asks this job to assert — + # come from the per-turn rubric observations, not from this generation, + # so a template narrative degrades the debrief rather than invalidating + # the run, and one unlucky generation must not red the nightly. + # + # But the debrief is generated exactly once, so a fallback means *no* + # model-written debrief prose was produced at all, and this is the only + # job that would ever notice. Say what the otherwise-green verdict + # therefore does not prove, rather than just naming the flag. + warnings.append( + "Debrief narrative used the deterministic fallback, not model prose: " + "the model's debrief output failed validation and the repair pass, so " + "this run exercised the scoring path but proves nothing about " + "real-model debrief generation. Recurring across nightlies means a " + "regression in the debrief prompt or DEBRIEF_NARRATIVE_SCHEMA rather " + "than one unlucky generation" + ) if not debrief.get("turning_points"): warnings.append("Debrief contains no turning points") return (failures, warnings) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index c5c09752..6694e2e1 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -481,7 +481,14 @@ def test_empty_summary_fails(self) -> None: def test_fallback_narrative_only_warns_because_scores_are_still_real(self) -> None: failures, warnings = smoke.evaluate_debrief(_debrief(used_fallback=True)) assert failures == [] - assert any("fallback" in w for w in warnings) + fallback = next(w for w in warnings if "fallback" in w) + # The debrief is generated once, so a fallback means no model-written + # debrief prose was produced at all and this is the only job that would + # notice. The run still passes -- the scores do not come from this + # generation -- so the warning has to say what the green verdict did not + # prove, or the limitation is invisible. + assert "proves nothing about real-model debrief generation" in fallback + assert "Recurring across nightlies" in fallback def test_absent_turning_points_only_warn(self) -> None: failures, warnings = smoke.evaluate_debrief(_debrief(turning_points=[])) From 9a823a504db90f0796e0183e7e7bdb391243d16f Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 23:28:46 -0400 Subject: [PATCH 29/58] Fail a nightly whose NPC turns only recite the authored opening MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every one of the last seven nightly runs on main logged an NPC answer to the player's first turn whose leading characters were byte-identical to the behavioral_interview scenario's `opening_npc_says` — the model reciting the opening question back instead of answering it. Nothing in the harness could see that. The per-turn parse flags are the stated proof that "a real model, not a canned fallback, drove the conversation", but an echo of the opening is neither empty nor SAFE_FALLBACK_UTTERANCE, so `used_fallback` stays false and the turn reads as healthy. The green verdict then rested on the reply merely being non-empty, which is not what #457 asks this job to assert. Record `replayed_opening` per turn and give it the same policy as the fallback check, for the same reason: one such turn degrades the run (a 4 B model is allowed an off sample) and warns, while *every* generated turn doing it means the model answered no player turn at all, so the run proves nothing and fails as `pipeline`. The comparison is a leading `startswith` on whitespace-collapsed text, because the behaviour observed in CI was a recital followed by fresh prose — equality would have missed it — and an unreadable opening convicts nobody. Verified against the real starter model (Qwen3 4B Q4_K_M, the registry pin) on a local llama.cpp llama-server: three real replies, none reciting the opening, `replayed_opening` false on every turn, scored debrief (4 dimensions, overall 49.8), run_smoke exit 0. The recital does not reproduce there, which is why the nightly — on llama-cpp-python's server and CI's conditions — is the only place it shows up, and why this check belongs in it. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 13 +- docs/real-model-smoke.md | 13 +- scripts/nightly-model-smoke.py | 65 +++++++- tests/scripts/test_nightly_model_smoke.py | 141 ++++++++++++++++++ 4 files changed, 226 insertions(+), 6 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 6a238aa3..1ff472ce 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -56,6 +56,17 @@ on a **real local model** end-to-end — registry download → llama.cpp → The NPC *opening* line is authored scenario text, not a generation — so the scripted player turns, not the opening, are what prove the model is working. +A turn that is real model output is not automatically a *reply*, and the +per-turn parse flags cannot tell the difference. Every nightly run of the +previous single-turn harness logged an NPC answer whose leading characters were +byte-identical to the scenario's `opening_npc_says` — the model reciting the +opening question back instead of answering it. `used_fallback` stays false for +those turns (the utterance is neither empty nor the canned safe one), so the run +went green on a conversation that never happened. The harness therefore records +`replayed_opening` per turn and applies the same policy it applies to +fallbacks: one such turn warns, *every* generated turn doing it fails, because +then the model answered nothing and the run proves nothing. + If the NPC closes the conversation before the script runs out (`session_control.continue_session`), the run still passes on the turns it did play — that is the product working as designed — but it records @@ -76,7 +87,7 @@ log, and a remedy printed next to it: | 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | | 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner. A port conflict started no child and so has no tail: the banner carries its own remedy instead, naming the port to free — see [Running it locally](#running-it-locally) | -| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | +| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect the per-turn `used_fallback` and `replayed_opening` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | | 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download or diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index bd85e5b2..bc2c3b41 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -52,6 +52,17 @@ on a **real local model** end-to-end — registry download → llama.cpp → The NPC *opening* line is authored scenario text, not a generation — so the scripted player turns, not the opening, are what prove the model is working. +A turn that is real model output is not automatically a *reply*, and the +per-turn parse flags cannot tell the difference. Every nightly run of the +previous single-turn harness logged an NPC answer whose leading characters were +byte-identical to the scenario's `opening_npc_says` — the model reciting the +opening question back instead of answering it. `used_fallback` stays false for +those turns (the utterance is neither empty nor the canned safe one), so the run +went green on a conversation that never happened. The harness therefore records +`replayed_opening` per turn and applies the same policy it applies to +fallbacks: one such turn warns, *every* generated turn doing it fails, because +then the model answered nothing and the run proves nothing. + If the NPC closes the conversation before the script runs out (`session_control.continue_session`), the run still passes on the turns it did play — that is the product working as designed — but it records @@ -72,7 +83,7 @@ log, and a remedy printed next to it: | 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | | 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner. A port conflict started no child and so has no tail: the banner carries its own remedy instead, naming the port to free — see [Running it locally](#running-it-locally) | -| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect per-turn `used_fallback` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | +| 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect the per-turn `used_fallback` and `replayed_opening` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | | 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download or diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index f351344f..a0082dea 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -290,8 +290,9 @@ class FailureClass: ), FailureClass.PIPELINE: ( "Both servers were healthy but an end-to-end assertion failed: the model " - "ran and produced output the product rejects. Inspect per-turn " - "used_fallback / native-structured-output flags in the report artifact." + "ran and produced output the product rejects. Inspect the per-turn " + "used_fallback, native-structured-output and replayed_opening flags in " + "the report artifact." ), FailureClass.TIMEOUT: ( "The run exhausted its wall-clock budget in the phase named above. " @@ -784,11 +785,36 @@ def _total_rubric_observations(turns: Sequence[Dict[str, Any]]) -> Optional[int] return None if nothing_readable else 0 +def _collapse(text: str) -> str: + """Normalise whitespace so two utterances can be compared as written.""" + return " ".join(text.split()) + + def _excerpt(text: str) -> str: - collapsed = " ".join(text.split()) + collapsed = _collapse(text) return collapsed[:EXCERPT_CHARS] + ("…" if len(collapsed) > EXCERPT_CHARS else "") +def _replays_opening(npc_text: str, opening_text: str) -> bool: + """True when an NPC reply opens by reciting the authored opening verbatim. + + The scenario's opening line sits in the transcript the turn prompt renders, + and the starter model answers the first player turn by copying it back: + every nightly run of the previous single-turn harness logged an NPC reply + whose leading characters matched ``opening_npc_says`` exactly. That is not + a reply, but nothing downstream notices — the utterance is non-empty and it + is not ``SAFE_FALLBACK_UTTERANCE``, so ``used_fallback`` stays false and the + per-turn parse flags report a healthy turn. Proving the turns are real + therefore needs this check as well as those flags. + + ``startswith`` rather than equality: a copy that then carries on into fresh + prose is the same regurgitation, and the leading recital is what identifies + it. No legitimate reply begins by reading the opening question back out. + """ + opening = _collapse(opening_text) + return bool(opening) and _collapse(npc_text).startswith(opening) + + # ── Assertions (pure, unit-tested) ──────────────────────────────────────────── @@ -808,6 +834,12 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: Turns carrying no ``used_fallback`` flag at all — the debug endpoint is best-effort — warn that the check did not run, so an absent proof is not mistaken for a passed one. + + ``replayed_opening`` gets the identical policy for the other way a turn can + pass every flag without being a reply: reciting the authored opening back at + the player (see ``_replays_opening``). Unlike the parse flags this is + computed from the response the harness already holds, so it is never + missing. """ failures: List[str] = [] warnings: List[str] = [] @@ -846,6 +878,26 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: f"{len(fallbacks)} of {len(generated)} model-generated NPC turns fell back " "to the safe utterance" ) + + # Same policy as the fallback check above, for the other way a turn can look + # real without being one: reciting the authored opening back instead of + # answering. See _replays_opening — the parse flags cannot see this, so + # without it "the model produced real NPC turns" rests on the utterance + # merely being non-empty. + echoes = [t for t in generated if t.get("replayed_opening")] + if echoes and len(echoes) == len(generated): + failures.append( + f"All {len(generated)} model-generated NPC turns began by reciting the " + "authored opening line verbatim; the model answered no player turn, so " + "this run proves nothing about real NPC replies" + ) + elif echoes: + warnings.append( + f"{len(echoes)} of {len(generated)} model-generated NPC turns began by " + "reciting the authored opening line verbatim instead of answering the " + "player: real output, but not a reply. Recurring in the same turn " + "position every night is a turn-prompt problem, not an unlucky sample" + ) return (failures, warnings) @@ -1347,13 +1399,18 @@ def run_smoke( latency_ms = (time.monotonic() - t0) * 1000 turn_latencies.append(latency_ms) npc_text = _npc_turn_content(turn.get("events", [])) - print(f" {latency_ms:.0f} ms | NPC: {_excerpt(npc_text)!r}") + replayed_opening = _replays_opening(npc_text, opening) + print(f" {latency_ms:.0f} ms | NPC: {_excerpt(npc_text)!r}" + + (" [replays the authored opening]" if replayed_opening else "")) results["turns"].append({ "label": f"player_turn_{i}", "turn_number": i, "model_generated": True, "latency_ms": round(latency_ms), "npc_excerpt": _excerpt(npc_text), + # Real model output that is not a reply — see _replays_opening. + # The parse flags below cannot tell this from a healthy turn. + "replayed_opening": replayed_opening, "ending_type": turn.get("ending_type"), # What the model volunteered for the debrief to score. Recorded # per turn so an unscored debrief can be attributed without diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 6694e2e1..d3e34e7d 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -364,6 +364,101 @@ def test_opening_only_run_fails(self) -> None: failures, _ = smoke.evaluate_turns(turns) assert any("No model-generated" in f for f in failures) + def test_every_turn_reciting_the_opening_fails(self) -> None: + # Real model output, every parse flag healthy, and not one reply: the + # exact run the fallback check cannot see. + turns = [ + _turn(replayed_opening=True), + _turn(turn_number=2, replayed_opening=True), + ] + failures, _ = smoke.evaluate_turns(turns) + assert any("reciting the authored opening" in f for f in failures) + + def test_one_turn_reciting_the_opening_only_warns(self) -> None: + # Observed behaviour on CI: the starter model recites the opening back + # on the first player turn. One such turn degrades the run without + # making it worthless, so it warns — the same policy as one fallback. + turns = [ + _turn(replayed_opening=True), + _turn(turn_number=2), + _turn(turn_number=3), + ] + failures, warnings = smoke.evaluate_turns(turns) + assert failures == [] + assert any("reciting the authored opening" in w for w in warnings) + + def test_a_recited_opening_is_not_mistaken_for_a_fallback(self) -> None: + # The two have different causes and different fixes, so the verdict + # must not describe one as the other. + failures, warnings = smoke.evaluate_turns([_turn(replayed_opening=True)]) + assert not any("fell back" in m for m in failures + warnings) + + def test_the_authored_opening_is_not_judged_for_reciting_itself(self) -> None: + turns = [ + {"label": "npc_opening", "turn_number": 0, "model_generated": False, + "npc_excerpt": "Thanks for coming in.", "replayed_opening": True}, + _turn(), + ] + failures, warnings = smoke.evaluate_turns(turns) + assert failures == [] + assert warnings == [] + + +class TestReplaysOpening: + """Reciting the authored opening is real model output that is not a reply. + + Every nightly run of the previous single-turn harness logged an NPC reply + whose leading characters were byte-identical to the scenario's + ``opening_npc_says``. ``used_fallback`` stays false for those turns — the + utterance is neither empty nor the canned safe one — so without this check + "the model produced real NPC turns" rests on the reply being non-empty. + """ + + OPENING = ( + "Thanks for coming in today. I'm Alex Chen from HR. Tell me a little " + "about yourself and why you're interested in this role." + ) + + def test_a_verbatim_copy_is_caught(self) -> None: + assert smoke._replays_opening(self.OPENING, self.OPENING) + + def test_a_copy_that_carries_on_into_fresh_prose_is_caught(self) -> None: + # What the nightly logs actually showed: the recital, then more text. + # Equality would have missed it. + assert smoke._replays_opening( + self.OPENING + " So, tell me about a time you shipped something.", + self.OPENING, + ) + + def test_reformatted_whitespace_does_not_hide_a_copy(self) -> None: + assert smoke._replays_opening( + "Thanks for coming in today.\n I'm Alex Chen from HR.\tTell me a " + "little\nabout yourself and why you're interested in this role.", + self.OPENING, + ) + + def test_a_real_reply_passes(self) -> None: + assert not smoke._replays_opening( + "Five years is a solid run. Walk me through the hardest trade-off " + "you made on that API platform.", + self.OPENING, + ) + + def test_a_reply_merely_echoing_a_few_words_passes(self) -> None: + # Only a leading recital of the whole opening counts; sharing an + # opening phrase does not. + assert not smoke._replays_opening( + "Thanks for coming in today was my line, not yours — but go on.", + self.OPENING, + ) + + def test_an_unreadable_opening_cannot_convict_a_turn(self) -> None: + # _npc_turn_content returns '' when the opening event is missing or + # malformed. Every reply startswith('') is True, so an absent opening + # would otherwise fail the whole conversation. + assert not smoke._replays_opening("A perfectly good reply.", "") + assert not smoke._replays_opening("A perfectly good reply.", " \n ") + # --------------------------------------------------------------------------- # Debrief assertions @@ -1496,6 +1591,52 @@ def _start_core(data_dir, *args, **kwargs) -> _FakeProc: assert seen["existed_at_terminate"] is True assert not seen["data_dir"].exists(), "the data directory leaked" + def test_a_conversation_of_recited_openings_does_not_pass( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + """Every flag healthy, every "reply" the authored opening read back. + + This is the run the parse flags cannot fail: the utterances are real + model output, non-empty and not the canned fallback. It must not report + that the model produced real NPC turns. + """ + models_dir, model_id, digest = staged_model + opening = _OPENING["events"][0]["payload"]["content"] + recital = {"events": [{"event_type": "npc_turn", "payload": { + "content": f"{opening} Now, tell me about a time you disagreed.", + "rubric_observations": [], + }}], "ending_type": None} + monkeypatch.setattr( + smoke, "_request_json", _fake_core(_debrief(), turn=recital) + ) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failed_phase"] == "assertions" + assert any("reciting the authored opening" in f for f in results["failures"]) + # Recorded per turn, so triage can see which turns did it without + # re-reading the excerpts. + assert all(t["replayed_opening"] for t in results["turns"][1:]) + + def test_a_healthy_run_records_that_no_turn_recited_the_opening( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core(_debrief())) + report = tmp_path / "report.json" + + assert smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) == 0 + results = json.loads(report.read_text(encoding="utf-8")) + assert all(t["replayed_opening"] is False for t in results["turns"][1:]) + assert results["warnings"] == [] + def test_unscored_debrief_is_a_pipeline_failure( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: From cd42a02efd6ed6e3477ea8c57ddf904554d0beaf Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 23:47:15 -0400 Subject: [PATCH 30/58] Catch a recital of the authored opening that breaks off partway The replayed_opening guard required an NPC reply to begin with the *entire* opening_npc_says before calling it a recital rather than a reply. The evidence it was built from cannot support that bar: the old harness logged `npc_text[:80]` against a 122-character opening, so the seven nightlies establish only that the reply's first 80 characters were the opening's first 80. Whether the copy ran all the way to the end of the authored line or veered into the model's own words after a sentence or two was never recorded -- and in the second case the check would never have fired, leaving a dead guard on exactly the behaviour it exists to catch while the nightly kept going green on a conversation that never happened. Match on a leading prefix of REPLAY_PREFIX_CHARS (40 collapsed characters) instead. Forty characters of the scenario's own text at the head of an answer is not reached by chance -- the transcript the turn prompt renders is the only place it can come from -- and the threshold is capped at the opening's own length, so a scenario with a terser opening keeps the strict whole-opening semantics and gains no new way to false-positive. The failure and warning text said "verbatim", which read as "the whole line"; it now says the turn recited the opening back at the player. Tests: a recital that breaks off partway is caught, a short opening still has to be recited in full, and an empty utterance (already its own failure) is not also convicted of reciting. Both copies of the doc explain the threshold and why it is not the whole opening. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 13 ++++++ docs/real-model-smoke.md | 13 ++++++ scripts/nightly-model-smoke.py | 42 +++++++++++++++---- tests/scripts/test_nightly_model_smoke.py | 32 +++++++++++++- 4 files changed, 90 insertions(+), 10 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 1ff472ce..93385986 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -67,6 +67,19 @@ went green on a conversation that never happened. The harness therefore records fallbacks: one such turn warns, *every* generated turn doing it fails, because then the model answered nothing and the run proves nothing. +What counts as a recital is a *leading prefix* match, `REPLAY_PREFIX_CHARS` +(40 collapsed characters) of the authored opening reproduced at the head of the +reply — not the whole opening. Those nightly logs printed only `npc_text[:80]` +against a 122-character opening, so what they establish is that the reply +*began* as the opening; whether the copy ran to the end or veered into the +model's own words after a sentence or two was never recorded, and a check that +demanded the entire line would have gone unreachable on the second case. Forty +characters of the scenario's own text at the head of an answer is not something +a reply arrives at by chance — the transcript the turn prompt renders is the +only place it can come from — and the threshold is capped at the opening's own +length, so a scenario with a terser opening still has to have it recited in +full. + If the NPC closes the conversation before the script runs out (`session_control.continue_session`), the run still passes on the turns it did play — that is the product working as designed — but it records diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index bc2c3b41..43012876 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -63,6 +63,19 @@ went green on a conversation that never happened. The harness therefore records fallbacks: one such turn warns, *every* generated turn doing it fails, because then the model answered nothing and the run proves nothing. +What counts as a recital is a *leading prefix* match, `REPLAY_PREFIX_CHARS` +(40 collapsed characters) of the authored opening reproduced at the head of the +reply — not the whole opening. Those nightly logs printed only `npc_text[:80]` +against a 122-character opening, so what they establish is that the reply +*began* as the opening; whether the copy ran to the end or veered into the +model's own words after a sentence or two was never recorded, and a check that +demanded the entire line would have gone unreachable on the second case. Forty +characters of the scenario's own text at the head of an answer is not something +a reply arrives at by chance — the transcript the turn prompt renders is the +only place it can come from — and the threshold is capped at the opening's own +length, so a scenario with a terser opening still has to have it recited in +full. + If the NPC closes the conversation before the script runs out (`session_control.continue_session`), the run still passes on the turns it did play — that is the product working as designed — but it records diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index a0082dea..47595edf 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -169,6 +169,26 @@ class exit meaning # A debrief summary shorter than this is not a usable coaching summary. MIN_SUMMARY_CHARS = 20 +# How much of the authored opening an NPC reply has to recite, leading, before +# we stop calling it a reply. See _replays_opening. +# +# Not "the whole opening": the evidence this check was built from is the old +# harness's log line, which printed `npc_text[:80]` against a 122-character +# opening, so what the nightlies actually establish is that the reply's first +# 80 characters were the opening's first 80. Whether the recital ran all the +# way to "…interested in this role." or veered off into the model's own words +# after the first sentence or two was never logged, and a check that demanded +# the entire opening would have quietly failed to fire on the latter — a dead +# guard on the exact behaviour it exists to catch. +# +# 40 collapsed characters is ~7 words of the scenario's own text reproduced +# verbatim at the head of a reply ("Thanks for coming in today. I'm Alex Che"), +# which no answer to a player turn arrives at by chance; the transcript the +# turn prompt renders is the only place it can come from. The threshold is +# capped at the opening's own length below, so a scenario whose opening is +# shorter than this keeps the strict whole-opening semantics. +REPLAY_PREFIX_CHARS = 40 + # Appended to the "no scores" failure. The debrief's dimension scores are # accumulated purely from rubric_observations the *model* volunteers, and # nothing asks it for them: the built-in behavioral_interview scenario defines @@ -807,12 +827,18 @@ def _replays_opening(npc_text: str, opening_text: str) -> bool: per-turn parse flags report a healthy turn. Proving the turns are real therefore needs this check as well as those flags. - ``startswith`` rather than equality: a copy that then carries on into fresh - prose is the same regurgitation, and the leading recital is what identifies - it. No legitimate reply begins by reading the opening question back out. + A *leading prefix* match, not equality and not the whole opening: a recital + that then carries on into fresh prose is the same regurgitation, and a + recital that veers off partway through the opening is too — see + ``REPLAY_PREFIX_CHARS`` for why the latter matters and where the length + comes from. No legitimate reply begins by reading the opening question + back out. """ opening = _collapse(opening_text) - return bool(opening) and _collapse(npc_text).startswith(opening) + if not opening: + return False + threshold = min(len(opening), REPLAY_PREFIX_CHARS) + return _collapse(npc_text)[:threshold] == opening[:threshold] # ── Assertions (pure, unit-tested) ──────────────────────────────────────────── @@ -888,14 +914,14 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: if echoes and len(echoes) == len(generated): failures.append( f"All {len(generated)} model-generated NPC turns began by reciting the " - "authored opening line verbatim; the model answered no player turn, so " - "this run proves nothing about real NPC replies" + "authored opening line back at the player; the model answered no player " + "turn, so this run proves nothing about real NPC replies" ) elif echoes: warnings.append( f"{len(echoes)} of {len(generated)} model-generated NPC turns began by " - "reciting the authored opening line verbatim instead of answering the " - "player: real output, but not a reply. Recurring in the same turn " + "reciting the authored opening line back at the player instead of " + "answering them: real output, but not a reply. Recurring in the same turn " "position every night is a turn-prompt problem, not an unlucky sample" ) return (failures, warnings) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index d3e34e7d..82bcf136 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -430,6 +430,19 @@ def test_a_copy_that_carries_on_into_fresh_prose_is_caught(self) -> None: self.OPENING, ) + def test_a_recital_that_veers_off_partway_is_caught(self) -> None: + # The evidence this check was built from is the old harness's log line, + # which printed npc_text[:80] against a 122-character opening: the + # nightlies establish the first 80 characters matched and say nothing + # about the rest. Demanding the entire opening would have made the + # guard silently unreachable for a recital that breaks off early, which + # is just as much "not a reply". + assert smoke._replays_opening( + "Thanks for coming in today. I'm Alex Chen from HR. Tell me a little " + "about your background in software development.", + self.OPENING, + ) + def test_reformatted_whitespace_does_not_hide_a_copy(self) -> None: assert smoke._replays_opening( "Thanks for coming in today.\n I'm Alex Chen from HR.\tTell me a " @@ -445,13 +458,28 @@ def test_a_real_reply_passes(self) -> None: ) def test_a_reply_merely_echoing_a_few_words_passes(self) -> None: - # Only a leading recital of the whole opening counts; sharing an - # opening phrase does not. + # Only a leading recital long enough to be the scenario's own text + # counts (REPLAY_PREFIX_CHARS); sharing an opening phrase does not. assert not smoke._replays_opening( "Thanks for coming in today was my line, not yours — but go on.", self.OPENING, ) + def test_a_short_opening_still_has_to_be_recited_in_full(self) -> None: + # The prefix threshold is capped at the opening's own length, so it + # only ever loosens the check for openings longer than it — a scenario + # with a terse opening keeps the strict whole-opening semantics and + # gains no new way to false-positive. + short = "Hello. Begin when ready." + assert len(short) < smoke.REPLAY_PREFIX_CHARS + assert smoke._replays_opening(short + " So, your background?", short) + assert not smoke._replays_opening("Hello. Five years in software.", short) + + def test_an_empty_reply_is_not_convicted_of_reciting(self) -> None: + # An empty utterance is already a failure in its own right + # (evaluate_turns); it must not also be reported as a recital. + assert not smoke._replays_opening("", self.OPENING) + def test_an_unreadable_opening_cannot_convict_a_turn(self) -> None: # _npc_turn_content returns '' when the opening event is missing or # malformed. Every reply startswith('') is True, so an absent opening From feaf6c05cf62771f1643a60fdf17efa2e09476ea Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 23:49:42 -0400 Subject: [PATCH 31/58] Say what a null rubric-observation total actually establishes _total_rubric_observations returns None whenever any generated turn's count is unreadable and the readable ones total zero -- counts of [0, None, 0] give None even though two turns were read. That is the right behaviour: reporting 0 would have the unscored-debrief verdict assert the model volunteered nothing across the conversation when the turn it could not read may have carried plenty. Three places described a narrower condition, so a triager reading them against the artifact would think the report contradicted itself: - the docstring said None happens "only when *nothing* readable was found", which is false for [0, None, 0]; - the "null" bullet in both copies of docs/real-model-smoke.md said "the NPC turn payloads carried no readable rubric_observations list", implying every per-turn rubric_observation_count would be null; - the failure message opened "the run could not read how many rubric observations the NPC turns carried", when it had read some of them. All three now say what None means -- at least one turn unreadable and the readable ones carried nothing, so the sum cannot be trusted as a total -- and point at the per-turn counts to find which turns were the unreadable ones. The local `nothing_readable` is renamed `zero_is_unreliable` after what it actually gates. The [0, None, 0] case is added to the parametrised test (the mirror-image [None, 0, 0] was already there) and its name no longer repeats the wrong claim. Reported by a concurrent review session. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 18 +++++---- docs/real-model-smoke.md | 18 +++++---- scripts/nightly-model-smoke.py | 39 ++++++++++++------- tests/scripts/test_nightly_model_smoke.py | 12 +++++- 4 files changed, 57 insertions(+), 30 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 93385986..c1d3b7a7 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -182,13 +182,17 @@ So an unscored debrief means one of three quite different things: that arrived pre-parsed from the runtime adapter — loses its observations on the way to the debrief. Compare `raw_npc_output` from the `/debug` payload against the per-turn `rubric_observation_count` in the artifact. -- **`rubric_observations_seen` is `null` — the harness could not tell.** The NPC - turn payloads carried no readable `rubric_observations` list, so the per-turn - `rubric_observation_count` is `null` too and the message quotes both causes - above rather than picking one. `convsim-core` always sends that list today, so - this is itself worth chasing: the turn response contract changed. The harness - deliberately does not score an unreadable payload as zero — that would have - the verdict name the one cause it has no evidence for. +- **`rubric_observations_seen` is `null` — the harness could not tell.** At + least one generated turn carried no readable `rubric_observations` list, and + the turns it *could* read carried nothing between them, so the total cannot be + trusted as a total and the message quotes both causes above rather than + picking one. Check `rubric_observation_count` per turn in the report to see + which turns were unreadable (`null`) and which really were zero; `null` is not + "every turn was unreadable". `convsim-core` always sends that list today, so + any `null` is itself worth chasing: the turn response contract changed. The + harness deliberately does not round an unreadable payload down to zero — that + would have the verdict assert the model volunteered nothing when the turn it + could not read may have carried plenty. In every case the run is red: *"scored debrief"* is the acceptance criterion for [#457](https://github.com/outrightmental/ConversationSimulator/issues/457). diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 43012876..f7ec0ffc 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -178,13 +178,17 @@ So an unscored debrief means one of three quite different things: that arrived pre-parsed from the runtime adapter — loses its observations on the way to the debrief. Compare `raw_npc_output` from the `/debug` payload against the per-turn `rubric_observation_count` in the artifact. -- **`rubric_observations_seen` is `null` — the harness could not tell.** The NPC - turn payloads carried no readable `rubric_observations` list, so the per-turn - `rubric_observation_count` is `null` too and the message quotes both causes - above rather than picking one. `convsim-core` always sends that list today, so - this is itself worth chasing: the turn response contract changed. The harness - deliberately does not score an unreadable payload as zero — that would have - the verdict name the one cause it has no evidence for. +- **`rubric_observations_seen` is `null` — the harness could not tell.** At + least one generated turn carried no readable `rubric_observations` list, and + the turns it *could* read carried nothing between them, so the total cannot be + trusted as a total and the message quotes both causes above rather than + picking one. Check `rubric_observation_count` per turn in the report to see + which turns were unreadable (`null`) and which really were zero; `null` is not + "every turn was unreadable". `convsim-core` always sends that list today, so + any `null` is itself worth chasing: the turn response contract changed. The + harness deliberately does not round an unreadable payload down to zero — that + would have the verdict assert the model volunteered nothing when the turn it + could not read may have carried plenty. In every case the run is red: *"scored debrief"* is the acceptance criterion for [#457](https://github.com/outrightmental/ConversationSimulator/issues/457). diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 47595edf..d404e823 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -788,12 +788,22 @@ def _rubric_observation_count(events: Sequence[dict]) -> Optional[int]: def _total_rubric_observations(turns: Sequence[Dict[str, Any]]) -> Optional[int]: """Total the per-turn rubric observation counts, or ``None`` if unknowable. - ``None`` only when *nothing* readable was found — an unreadable payload, or - no generated turn to read at all. A total above zero already settles which - cause of an unscored debrief applies (the turns did carry observations), so - an unreadable turn alongside readable ones does not make the verdict - ambiguous. Only the authored opening is skipped: it is not a generation and - so has no count to miss. + ``None`` means *the sum cannot be trusted as a total*, which is not the same + as "nothing was readable": + + * a total above zero is conclusive even with an unreadable turn among the + readable ones — observations *were* seen, so the debrief losing them is + the cause whatever the missing turn carried; + * a sum of zero is only conclusive when every generated turn was readable. + With counts of ``[0, None, 0]`` two turns really did carry nothing, but + reporting 0 would have the verdict assert the model volunteered nothing + across the conversation when the turn it could not read may have carried + plenty — so that case is ``None`` too, and the report's per-turn + ``rubric_observation_count`` shows which turn was the unreadable one; + * no generated turn at all is ``None``: there was nothing to read. + + Only the authored opening is skipped: it is not a generation and so has no + count to miss. """ counts = [ t.get("rubric_observation_count") for t in turns if t.get("model_generated") @@ -801,8 +811,8 @@ def _total_rubric_observations(turns: Sequence[Dict[str, Any]]) -> Optional[int] total = sum(c for c in counts if isinstance(c, int)) if total > 0: return total - nothing_readable = not counts or any(not isinstance(c, int) for c in counts) - return None if nothing_readable else 0 + zero_is_unreliable = not counts or any(not isinstance(c, int) for c in counts) + return None if zero_is_unreliable else 0 def _collapse(text: str) -> str: @@ -976,12 +986,13 @@ def evaluate_debrief( # that just said it could not tell, is the triage this harness is # supposed to have done, done wrongly. cause = ( - "the run could not read how many rubric observations the NPC " - "turns carried, so the two causes below cannot be told apart " - "from this message — read rubric_observation_count per turn in " - "the report artifact (null means the turn payload carried no " - "readable rubric_observations list, which is itself worth a " - "look at the turn response contract). " + UNSCORED_DEBRIEF_NOTE + "at least one NPC turn's rubric_observations list was " + "unreadable and the turns that were readable carried none, so " + "the run cannot claim the model volunteered nothing and the two " + "causes below cannot be told apart from this message — read " + "rubric_observation_count per turn in the report artifact (null " + "marks the unreadable turns, which is itself worth a look at the " + "turn response contract). " + UNSCORED_DEBRIEF_NOTE + " If the turns did carry observations, it is the other cause " "instead: " + UNSCORED_WITH_OBSERVATIONS_NOTE ) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 82bcf136..6baa0052 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -1074,11 +1074,16 @@ def test_an_unreadable_observation_list_is_unknown_not_zero( ([1, 0, 2], 3), ([None, None, None], None), # nothing readable anywhere ([None, 0, 0], None), # a gap with no positive evidence + ([0, None, 0], None), # ...wherever in the run the gap falls ([None, 2, 0], 2), # observations were seen regardless ]) - def test_the_total_is_unknown_only_when_nothing_was_readable( + def test_a_zero_total_is_reported_only_when_every_turn_was_readable( self, counts: list, expected: int | None ) -> None: + # None does not mean "nothing was readable": two readable zeroes + # alongside one unreadable turn is still None, because reporting 0 would + # have the verdict assert the model volunteered nothing across the + # conversation when the unread turn may have carried plenty. turns = [{"model_generated": False, "label": "npc_opening"}] + [ {"model_generated": True, "rubric_observation_count": c} for c in counts ] @@ -1762,7 +1767,10 @@ def test_a_turn_payload_without_observations_is_not_read_as_zero( t["rubric_observation_count"] is None for t in results["turns"][1:] ), "a null per-turn count is what tells the reader the list was unreadable" no_scores = next(f for f in results["failures"] if "rubric dimension" in f) - assert "could not read" in no_scores + # Names what it actually observed (an unreadable list) and keeps both + # causes open, rather than asserting the one it has no evidence for. + assert "unreadable" in no_scores + assert "cannot be told apart" in no_scores assert smoke.UNSCORED_DEBRIEF_NOTE in no_scores assert smoke.UNSCORED_WITH_OBSERVATIONS_NOTE in no_scores From 280345ddeb610bce8beac1651a0a05fd9442e821 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Fri, 2 Oct 2026 23:56:38 -0400 Subject: [PATCH 32/58] Stop a source build of llama.cpp from eating the 30 min job budget MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PyPI publishes llama-cpp-python as an sdist only — the prebuilt CPU wheels live on the abetlen extra index, which is why this step uses one — but --extra-index-url does not prefer wheels over sources. pip resolves the newest version across both indexes, so on any night when the wheel index has not yet published the latest release, this step compiles llama.cpp on the runner instead of installing a wheel. That was survivable under the old timeout-minutes: 60. It is not under 30: a 15+ min build turns the ~2 min `pip install` row of the documented runtime budget into most of the job, and the nightly dies with exactly the unattributable "The operation was canceled" that the harness's phase- attributed 20 min deadline exists to replace — on a night when nothing about the product changed. --only-binary llama-cpp-python makes pip fall back to the newest version that does have a wheel, and if no wheel is available at all it fails in seconds against this step's own name rather than at the job ceiling. Co-Authored-By: Claude Opus 5 --- .github/workflows/model-smoke-nightly.yml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.github/workflows/model-smoke-nightly.yml b/.github/workflows/model-smoke-nightly.yml index 62b568e2..ff65188d 100644 --- a/.github/workflows/model-smoke-nightly.yml +++ b/.github/workflows/model-smoke-nightly.yml @@ -120,9 +120,21 @@ jobs: - name: Install convsim-core run: pip install -e "services/convsim-core[dev]" + # --only-binary enforces what the step name already promises. PyPI ships + # llama-cpp-python as an sdist only — the prebuilt wheels live on the + # extra index — and --extra-index-url does not prefer them: pip resolves + # the newest version across both indexes, so a release the wheel index has + # not caught up with yet is installed by compiling llama.cpp on the + # runner. That build is the ~2 min row in the runtime budget turning into + # 15+, which the 30 min job timeout now has no room for: the job would die + # with the unattributable cancellation this whole design exists to avoid, + # on a night when nothing about the product changed. Refusing the sdist + # makes pip fall back to the newest version that does have a wheel, and if + # there is none it fails in seconds against this step's own name. - name: Install llama-cpp-python with server extra (CPU-only prebuilt wheel) run: | pip install "llama-cpp-python[server]" \ + --only-binary llama-cpp-python \ --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu # Single source of truth: the cache key, the download URL and the From 149a7f7f2d0a22d13adc2e7b37cf4d6b12d53a81 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 00:00:33 -0400 Subject: [PATCH 33/58] Stop a case-shifted digest from deleting the model it matches `verify_model_checksum` compared `--model-sha256` to the computed digest as a string, so the same hash in upper case -- or with a stray newline from a paste -- read as drift. The consequences are out of all proportion to the typo: the file is deleted (2.5 GB, re-downloaded next run) and the verdict reports that the pinned upstream file was replaced and that the check must not be relaxed, sending whoever typed it looking for a supply-chain problem. The registry pins lower-case hex so CI never hits this, but the local repro in docs/real-model-smoke.md asks a human to pass the hash by hand. Compare it as a hex value instead: strip and lower-case `expected` before the comparison. Nothing is weakened -- a digest that differs in any nibble still fails, and the per-PR registry test still requires the pin itself to be lower-case. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 9 +++++++++ tests/scripts/test_nightly_model_smoke.py | 15 +++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index d404e823..07a29852 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -549,6 +549,14 @@ def verify_model_checksum(path: Path, expected: str, *, delete_on_mismatch: bool A mismatch deletes the file by default: leaving a known-bad GGUF in the CI cache would make every subsequent run fail on the same corrupt bytes, since the cache key is the expected hash and a cache hit skips the download. + + ``expected`` is compared as a hex *value*, not as a string: a digest that + differs only in case or surrounding whitespace is the same digest, and + string equality would call it drift. That matters because of what a + mismatch does — it deletes a 2.5 GB file and reports "the pinned upstream + file was replaced … do NOT relax this check", which is an expensive and + thoroughly misleading verdict to hand someone who pasted an upper-case + hash into the local repro in docs/real-model-smoke.md. """ if not path.exists(): raise SmokeFailure( @@ -556,6 +564,7 @@ def verify_model_checksum(path: Path, expected: str, *, delete_on_mismatch: bool f"Model file not found: {path}. The cache restore produced no model and no " "download ran — check the cache key and the download step.", ) + expected = expected.strip().lower() actual = sha256_file(path) if actual != expected: size_mb = path.stat().st_size // 1_048_576 diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 6baa0052..7d10ce5f 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -102,6 +102,21 @@ def test_matching_checksum_returns_the_digest(self, tmp_path: Path) -> None: assert smoke.verify_model_checksum(path, digest) == digest assert path.exists() + def test_the_same_digest_in_a_different_case_is_not_drift(self, tmp_path: Path) -> None: + # The registry pins lower-case hex, but --model-sha256 is typed by hand + # in the local repro. An upper-case paste is the same digest, and + # string equality would call it drift -- deleting a 2.5 GB file and + # reporting "the pinned upstream file was replaced", which is both + # expensive and wrong. + path, digest = self._write(tmp_path) + assert smoke.verify_model_checksum(path, digest.upper()) == digest + assert path.exists() + + def test_a_digest_pasted_with_stray_whitespace_is_not_drift(self, tmp_path: Path) -> None: + path, digest = self._write(tmp_path) + assert smoke.verify_model_checksum(path, f" {digest}\n") == digest + assert path.exists() + def test_drift_raises_checksum_class(self, tmp_path: Path) -> None: path, _ = self._write(tmp_path) with pytest.raises(smoke.SmokeFailure) as exc_info: From 9b4a32b4181f589f6fa3998e65270214968a9391 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 00:01:38 -0400 Subject: [PATCH 34/58] Keep the local repro off the source build the job now refuses 280345d added `--only-binary llama-cpp-python` to the workflow's install step, because `--extra-index-url` does not stop pip picking a PyPI sdist and compiling llama.cpp on the runner. The repro block in docs/real-model-smoke.md is a hand-made copy of that step and was left a flag behind, so the one procedure a reader follows to reproduce a red nightly is the one that can spend 15+ minutes building the wheel the job declines to build -- and concludes the harness is at fault. Added the flag to both copies of the doc. Pinned the promise in tests/scripts, next to the checksum invariants, for the same reason those exist: it lives in the YAML alone, it reads as a redundant flag beside the extra index that is "obviously" already doing the job, and nothing else in the suite would notice it going away. The test covers the workflow step and both doc copies. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 1 + docs/real-model-smoke.md | 1 + tests/scripts/test_nightly_model_smoke.py | 39 +++++++++++++++++++ 3 files changed, 41 insertions(+) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index c1d3b7a7..286490c7 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -307,6 +307,7 @@ Nothing here touches the network except the one-time model download. pip install -e "packages/prompt-composer[dev]" pip install -e "services/convsim-core[dev]" pip install "llama-cpp-python[server]" \ + --only-binary llama-cpp-python \ --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu # 2. Resolve the starter model's id / url / sha256 from the registry diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index f7ec0ffc..5f454912 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -303,6 +303,7 @@ Nothing here touches the network except the one-time model download. pip install -e "packages/prompt-composer[dev]" pip install -e "services/convsim-core[dev]" pip install "llama-cpp-python[server]" \ + --only-binary llama-cpp-python \ --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu # 2. Resolve the starter model's id / url / sha256 from the registry diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 7d10ce5f..fb2692ae 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -907,6 +907,45 @@ def test_the_restored_and_saved_cache_entries_are_the_same(self) -> None: assert restore["with"]["path"] == save["with"]["path"] +_REAL_MODEL_SMOKE_DOCS = ( + REPO_ROOT / "docs" / "real-model-smoke.md", + REPO_ROOT / "docs-site" / "src" / "content" / "docs" / "dev" / "real-model-smoke.md", +) + + +class TestWheelInstallInvariants: + """llama-cpp-python must arrive as a prebuilt wheel, never as a source build. + + PyPI ships llama-cpp-python as an sdist only — the CPU wheels live on the + extra index — and ``--extra-index-url`` does not *prefer* them: pip resolves + the newest version across both indexes, so on any night the wheel index lags + PyPI, pip compiles llama.cpp on the runner. That is the ~2 min install row + of the runtime budget turning into 15+, which the 30 min job timeout has no + room for: the job dies with exactly the unattributable "operation was + canceled" that the harness's phase-attributed deadline exists to replace, on + a night when nothing about the product changed. + + Like the checksum promises above, this lives in the YAML and nowhere else, + and it reads as a redundant flag next to the extra index that is "obviously" + already doing the job. + """ + + def test_the_llama_wheel_is_never_built_from_source(self) -> None: + run = _step_running("llama-cpp-python[server]")["run"] + assert "--only-binary llama-cpp-python" in run, ( + "--extra-index-url alone does not stop pip from picking a PyPI " + "sdist and compiling llama.cpp on the runner, which costs more " + "than the whole job budget allows" + ) + + @pytest.mark.parametrize("doc", _REAL_MODEL_SMOKE_DOCS, ids=lambda p: p.parts[0]) + def test_the_documented_local_repro_installs_the_same_wheel(self, doc: Path) -> None: + # The repro block is a hand-made copy of the install step, so it drifts + # silently. A reader who follows it and gets a 15 min source build + # concludes the harness is the problem. + assert "--only-binary llama-cpp-python" in doc.read_text(encoding="utf-8") + + class TestDeadline: def test_phase_durations_are_recorded(self) -> None: clock = smoke.Deadline(60.0) From 5a57aa553db14de7de3d04f7b966a8bab2849ade Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 00:04:02 -0400 Subject: [PATCH 35/58] Stop a talkative child from turning a classified failure into a traceback The child stderr tails were bare deques, appended to by a draining thread for as long as the child lives and iterated by `_dump_stderr_tails` from `run_smoke`'s *except* handler -- which runs before the `finally` block stops either child. Reader and writer therefore overlap for real, and iterating a deque another thread is appending to raises `RuntimeError: deque mutated during iteration` (verified: it fires within a few thousand iterations). From there it escapes the handler that had just assembled the verdict, so a reportable `timeout` or `pipeline` failure is re-reported by main()'s catch-all as "a bug in the smoke harness" -- the same failure mode as the `finally`-block raise out of `Deadline.finish()` fixed earlier in this branch. And it would fire on precisely the failures where the child is still talking: a timeout whose abandoned generation is still running, a 5xx being logged. Replaced the deques with a small `StderrTail` that appends and snapshots under a lock, and had the dump print from one snapshot so the header cannot promise more lines than it goes on to show. Added coverage for the tail and for `_drain`, neither of which had any. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 58 ++++++++++++++---- tests/scripts/test_nightly_model_smoke.py | 73 +++++++++++++++++++++++ 2 files changed, 120 insertions(+), 11 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 07a29852..f2017254 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1203,13 +1203,45 @@ def _write_report(report_path: Optional[Path], results: Dict[str, Any]) -> None: # ── Smoke run ───────────────────────────────────────────────────────────────── -def _drain(stream: object, sink: Optional[deque] = None) -> None: +class StderrTail: + """The most recent lines of a child's stderr, safe to read while it writes. + + A bare ``deque`` is the obvious container and the wrong one. The draining + thread keeps appending for as long as the child is alive, and the tails are + dumped from ``run_smoke``'s *except* handler — before the ``finally`` block + stops either child — so the reader and the writer genuinely overlap. + Iterating a deque another thread is appending to raises + ``RuntimeError: deque mutated during iteration``, which would escape the + handler and throw away the classified verdict it had just assembled: the + same way an already-exhausted ``Deadline.finish()`` once turned a + reportable timeout into a bare traceback. And it would do it on exactly + the failures where the child is still talking — a timeout whose abandoned + generation is still running, a 5xx being logged. + + So appends and reads take a lock, and readers get a snapshot rather than a + live view of the buffer. + """ + + def __init__(self, maxlen: int = 200) -> None: + self._lines: deque = deque(maxlen=maxlen) + self._lock = threading.Lock() + + def append(self, line: str) -> None: + with self._lock: + self._lines.append(line) + + def snapshot(self) -> List[str]: + with self._lock: + return list(self._lines) + + +def _drain(stream: object, sink: Optional[StderrTail] = None) -> None: """Consume a subprocess pipe in the background to avoid a full-buffer deadlock. - When ``sink`` (a bounded deque) is given, the most recent lines are retained - so they can be surfaced if the smoke fails. The child's stderr is otherwise - discarded, which makes a server-side 500 undiagnosable from the CI logs — the - client only ever sees ``HTTPError: 500`` with no server traceback. + When ``sink`` is given, the most recent lines are retained so they can be + surfaced if the smoke fails. The child's stderr is otherwise discarded, + which makes a server-side 500 undiagnosable from the CI logs — the client + only ever sees ``HTTPError: 500`` with no server traceback. """ try: for raw in stream: # type: ignore[attr-defined] @@ -1220,11 +1252,15 @@ def _drain(stream: object, sink: Optional[deque] = None) -> None: pass -def _dump_stderr_tails(tails: Sequence[tuple[str, deque]]) -> None: +def _dump_stderr_tails(tails: Sequence[tuple[str, StderrTail]]) -> None: for label, tail in tails: - if tail: - print(f"\n[smoke] ── {label} stderr (last {len(tail)} lines) ──", file=sys.stderr) - for line in tail: + # One snapshot, used for both the count and the lines: reading the + # length separately would let the header promise more lines than the + # dump prints while the child is still writing. + lines = tail.snapshot() + if lines: + print(f"\n[smoke] ── {label} stderr (last {len(lines)} lines) ──", file=sys.stderr) + for line in lines: print(f" {label[:4]}| {line}", file=sys.stderr) @@ -1316,8 +1352,8 @@ def run_smoke( model_path = models_dir / f"{model_id}.gguf" llama_proc: Optional[subprocess.Popen] = None core_proc: Optional[subprocess.Popen] = None - llama_stderr_tail: deque = deque(maxlen=200) - core_stderr_tail: deque = deque(maxlen=200) + llama_stderr_tail = StderrTail() + core_stderr_tail = StderrTail() # convsim-core's throwaway data directory (SQLite db, WAL, logs). Created # and removed by hand rather than with a `with TemporaryDirectory(...)` diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index fb2692ae..a500f79b 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -2430,6 +2430,79 @@ def test_a_usage_error_is_not_mistaken_for_a_download_failure(self) -> None: assert exc_info.value.code != smoke.EXIT_CODES[smoke.FailureClass.DOWNLOAD] +# --------------------------------------------------------------------------- +# Child stderr tails +# --------------------------------------------------------------------------- + + +class _LiveTail: + """A tail that raises when iterated, the way a deque does mid-append. + + ``_dump_stderr_tails`` runs from ``run_smoke``'s except handler, before the + finally block stops either child, so the draining thread is still appending. + Iterating the live buffer therefore has to be impossible, not merely + unlikely -- a ``RuntimeError`` from there escapes the handler and discards + the classified verdict. + """ + + def __init__(self, *lines: str) -> None: + self._lines = list(lines) + + def __iter__(self): + raise RuntimeError("deque mutated during iteration") + + def __len__(self) -> int: + raise RuntimeError("deque mutated during iteration") + + def snapshot(self) -> list: + return list(self._lines) + + +class TestStderrTails: + """The dump must not iterate a buffer another thread is writing to.""" + + def test_the_dump_reads_a_snapshot_not_the_live_buffer( + self, capsys: pytest.CaptureFixture + ) -> None: + smoke._dump_stderr_tails((("convsim-core", _LiveTail("boom", "trace")),)) + err = capsys.readouterr().err + assert "last 2 lines" in err + assert "boom" in err and "trace" in err + + def test_an_empty_tail_prints_no_header(self, capsys: pytest.CaptureFixture) -> None: + smoke._dump_stderr_tails((("llama-server", smoke.StderrTail()),)) + assert capsys.readouterr().err == "" + + def test_the_tail_keeps_only_the_most_recent_lines(self) -> None: + tail = smoke.StderrTail(maxlen=3) + for i in range(10): + tail.append(f"line {i}") + assert tail.snapshot() == ["line 7", "line 8", "line 9"] + + def test_a_snapshot_does_not_alias_the_buffer(self) -> None: + tail = smoke.StderrTail() + tail.append("first") + snapshot = tail.snapshot() + tail.append("second") + assert snapshot == ["first"] + + def test_draining_a_byte_stream_decodes_and_strips(self) -> None: + tail = smoke.StderrTail() + smoke._drain(iter([b"warning: slow\n", b"\xff bad utf-8\n"]), tail) + assert tail.snapshot() == ["warning: slow", "� bad utf-8"] + + def test_a_stream_that_breaks_mid_read_does_not_kill_the_drain_thread(self) -> None: + # The drain runs in a daemon thread with nothing to catch for it, and a + # pipe closed under it while the child is being terminated is routine. + def _broken(): + yield b"last words\n" + raise ValueError("pipe closed") + + tail = smoke.StderrTail() + smoke._drain(_broken(), tail) + assert tail.snapshot() == ["last words"] + + # --------------------------------------------------------------------------- # Log ordering # --------------------------------------------------------------------------- From aaa2aa1a422b96a14c302ff22ec347efd787958f Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 00:16:37 -0400 Subject: [PATCH 36/58] Stop a conversation of fallbacks from reading as a rubric-prompting gap Found by driving run_smoke end-to-end against a real convsim-core on the `llama_cpp` runtime with a stub OpenAI server returning unparseable turn output -- the failure a 4 B model on a CPU-only runner actually produces. Every turn fell back to the canned safe utterance, every turn therefore carried an empty `rubric_observations` list, and the debrief scored nothing. Both failures land in the same verdict, and the second one said: "an empty array is not the normal outcome ... check what changed about what reaches the model (the OUTPUT_SCHEMA layer, the starter model pin, sampling)" None of that is the thing to read. A turn that fell back carries no model output at all, so its empty observations list says nothing about rubric prompting -- the debrief would have had nothing to accumulate with a rubric layer in place. The run had the evidence in hand (`used_fallback` on every generated turn) and still sent triage after a different investigation from the one the fallback failure beside it was already pointing at. Same shape as the two unscored-debrief corrections earlier on this branch: the verdict asserting a cause it has not established. `evaluate_debrief` now takes `turns_all_fell_back` and, for a zero that the fallbacks explain, says the unscored debrief is downstream of them and points at the per-turn `parse_events` instead. The other three causes are unchanged. Both verdicts read "all fell back" through one `_all_generated_turns_fell_back` helper, so they cannot drift into a verdict carrying both notes, each naming the other as the wrong place to start. Absent parse flags still mean no fallback, so an unreadable debug endpoint keeps the old diagnosis rather than gaining a confident new one. Docs: the "Unscored debrief" section listed three causes; it lists four now, in both copies. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 24 +++-- docs/real-model-smoke.md | 24 +++-- scripts/nightly-model-smoke.py | 75 ++++++++++++++-- tests/scripts/test_nightly_model_smoke.py | 88 +++++++++++++++++++ 4 files changed, 190 insertions(+), 21 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 286490c7..b4fed540 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -163,10 +163,21 @@ across two runs volunteered at least one observation, and the debrief scored three to four dimensions with an `overall_score` near 50. Treat zero observations as a signal, not as the resting state. -So an unscored debrief means one of three quite different things: - -- **`rubric_observations_seen` is 0 — the model volunteered nothing.** Only the - prompt's bare schema hint was ever asking, so this is reachable by design — +So an unscored debrief means one of four quite different things: + +- **`rubric_observations_seen` is 0 and every generated turn fell back.** Then + this failure is downstream of the fallback failure reported beside it and + carries no independent signal: a turn that fell back returns the canned safe + utterance and an empty `rubric_observations` list whatever the prompt asked + for, so the debrief would have had nothing to accumulate even with a rubric + layer in place. Start from the per-turn `parse_events` in the `/debug` + payload — they say why the model's output was rejected — and judge the + scoring path on the next run. The harness detects this case and says so + rather than printing the next bullet's note, which would be the wrong + investigation. +- **`rubric_observations_seen` is 0 with turns that did not fall back — the + model volunteered nothing.** Only the prompt's bare schema hint was ever + asking, so this is reachable by design — but since the starter model does normally answer that hint, first check what changed about what reaches the model: the `OUTPUT_SCHEMA` prompt layer, the registry's `starter` pin, the adapter's JSON-schema constraint, sampling @@ -185,8 +196,9 @@ So an unscored debrief means one of three quite different things: - **`rubric_observations_seen` is `null` — the harness could not tell.** At least one generated turn carried no readable `rubric_observations` list, and the turns it *could* read carried nothing between them, so the total cannot be - trusted as a total and the message quotes both causes above rather than - picking one. Check `rubric_observation_count` per turn in the report to see + trusted as a total and the message quotes the two causes it cannot tell + apart — volunteered nothing, or volunteered something the debrief lost — + rather than picking one. Check `rubric_observation_count` per turn in the report to see which turns were unreadable (`null`) and which really were zero; `null` is not "every turn was unreadable". `convsim-core` always sends that list today, so any `null` is itself worth chasing: the turn response contract changed. The diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 5f454912..0e653116 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -159,10 +159,21 @@ across two runs volunteered at least one observation, and the debrief scored three to four dimensions with an `overall_score` near 50. Treat zero observations as a signal, not as the resting state. -So an unscored debrief means one of three quite different things: - -- **`rubric_observations_seen` is 0 — the model volunteered nothing.** Only the - prompt's bare schema hint was ever asking, so this is reachable by design — +So an unscored debrief means one of four quite different things: + +- **`rubric_observations_seen` is 0 and every generated turn fell back.** Then + this failure is downstream of the fallback failure reported beside it and + carries no independent signal: a turn that fell back returns the canned safe + utterance and an empty `rubric_observations` list whatever the prompt asked + for, so the debrief would have had nothing to accumulate even with a rubric + layer in place. Start from the per-turn `parse_events` in the `/debug` + payload — they say why the model's output was rejected — and judge the + scoring path on the next run. The harness detects this case and says so + rather than printing the next bullet's note, which would be the wrong + investigation. +- **`rubric_observations_seen` is 0 with turns that did not fall back — the + model volunteered nothing.** Only the prompt's bare schema hint was ever + asking, so this is reachable by design — but since the starter model does normally answer that hint, first check what changed about what reaches the model: the `OUTPUT_SCHEMA` prompt layer, the registry's `starter` pin, the adapter's JSON-schema constraint, sampling @@ -181,8 +192,9 @@ So an unscored debrief means one of three quite different things: - **`rubric_observations_seen` is `null` — the harness could not tell.** At least one generated turn carried no readable `rubric_observations` list, and the turns it *could* read carried nothing between them, so the total cannot be - trusted as a total and the message quotes both causes above rather than - picking one. Check `rubric_observation_count` per turn in the report to see + trusted as a total and the message quotes the two causes it cannot tell + apart — volunteered nothing, or volunteered something the debrief lost — + rather than picking one. Check `rubric_observation_count` per turn in the report to see which turns were unreadable (`null`) and which really were zero; `null` is not "every turn was unreadable". `convsim-core` always sends that list today, so any `null` is itself worth chasing: the turn response contract changed. The diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index f2017254..7dbe7505 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -250,6 +250,28 @@ class exit meaning "report artifact." ) +# The third cause, and the one where the zero says nothing about rubrics at all. +# A turn that fell back carries the canned safe utterance and an empty +# rubric_observations list by construction, so a conversation in which *every* +# generated turn fell back reaches the debrief with nothing to score no matter +# how well the rubric were prompted. UNSCORED_DEBRIEF_NOTE would send triage +# after "the OUTPUT_SCHEMA layer, the starter model pin, sampling" — a different +# investigation from the one the fallback failure in the same verdict is already +# pointing at — and its "an empty array is not the normal outcome" is beside the +# point when no array was ever produced. evaluate_turns has already failed the +# run for the fallbacks; this note says the unscored debrief is downstream of +# that and carries no independent signal. +UNSCORED_AFTER_FALLBACK_NOTE = ( + "This is a consequence of the fallback failure above, not a second finding: " + "a turn that fell back to the canned safe utterance carries an empty " + "rubric_observations list whatever the prompt asked for, so with every " + "generated turn falling back the debrief had nothing to accumulate and would " + "have had nothing to accumulate even with a rubric layer in place. Fix the " + "fallbacks first — the per-turn parse_events in the /debug payload say why " + "the model's output was rejected — and judge the scoring path on the next " + "run. Do not start from the rubric prompting; this run says nothing about it." +) + # How much slower the debrief generation is allowed to be than one NPC turn # before we stop calling it slow and start calling it hung. Only used to size # request timeouts (see run_smoke) — debrief latency itself is not budget-checked. @@ -863,6 +885,24 @@ def _replays_opening(npc_text: str, opening_text: str) -> bool: # ── Assertions (pure, unit-tested) ──────────────────────────────────────────── +def _all_generated_turns_fell_back(turns: Sequence[Dict[str, Any]]) -> bool: + """True when every model-generated turn returned the canned safe utterance. + + Two verdicts turn on this and they must agree: ``evaluate_turns`` fails the + run for it, and ``evaluate_debrief`` uses it to say that an unscored debrief + is downstream of those fallbacks rather than evidence about rubric + prompting. One of them deciding "all of them" while the other decides "not + quite all" would put both notes in the same verdict, each telling the reader + the other is the wrong place to start. + + False when there are no generated turns (``evaluate_turns`` reports that + separately) and when the parse flags were unavailable, since an absent + ``used_fallback`` is not a fallback. + """ + generated = [t for t in turns if t.get("model_generated")] + return bool(generated) and all(t.get("used_fallback") for t in generated) + + def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: """Check the conversation really came from the model. Returns (failures, warnings). @@ -913,7 +953,7 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: ) fallbacks = [t for t in generated if t.get("used_fallback")] - if len(fallbacks) == len(generated): + if _all_generated_turns_fell_back(turns): failures.append( f"All {len(generated)} model-generated NPC turns fell back to the canned " "safe utterance; the real model produced no usable turn output" @@ -950,20 +990,26 @@ def evaluate_debrief( debrief: Optional[Dict[str, Any]], *, rubric_observations_seen: Optional[int] = None, + turns_all_fell_back: bool = False, ) -> tuple[List[str], List[str]]: """Check that a *scored* debrief was produced. Returns (failures, warnings). "Scored" is the #457 acceptance criterion, so an unscored debrief is a - failure — but on its own that failure does not say which of two very + failure — but on its own that failure does not say which of several very different things happened, and the remedy differs completely. ``rubric_observations_seen`` (how many rubric observations the NPC turns actually returned, ``None`` when the run could not read any — see - ``_total_rubric_observations``) resolves it: - - * **0** — the model volunteered nothing to score, which nothing asked it - to. Reachable because of the thin prompt coverage, but not the normal - outcome (the real starter model volunteers observations on every turn), - so still worth chasing: ``UNSCORED_DEBRIEF_NOTE``. + ``_total_rubric_observations``) and ``turns_all_fell_back`` resolve it: + + * **0, every generated turn fell back** — the turns carried no observations + because they carried no model output at all. Not evidence about rubric + prompting; a consequence of the fallback failure ``evaluate_turns`` + already reported: ``UNSCORED_AFTER_FALLBACK_NOTE``. + * **0, turns that did not fall back** — the model volunteered nothing to + score, which nothing asked it to. Reachable because of the thin prompt + coverage, but not the normal outcome (the real starter model volunteers + observations on every turn), so still worth chasing: + ``UNSCORED_DEBRIEF_NOTE``. * **> 0** — the turns carried observations and the debrief lost them. A real regression with a named starting point: ``UNSCORED_WITH_OBSERVATIONS_NOTE``. @@ -984,6 +1030,12 @@ def evaluate_debrief( "observation(s), so the debrief engine had something to accumulate " "and did not. " ) + UNSCORED_WITH_OBSERVATIONS_NOTE + elif rubric_observations_seen == 0 and turns_all_fell_back: + cause = ( + "no NPC turn carried a rubric_observation, so the debrief engine " + "had nothing to accumulate — but none of them carried any model " + "output either. " + ) + UNSCORED_AFTER_FALLBACK_NOTE elif rubric_observations_seen == 0: cause = ( "no NPC turn carried a rubric_observation, so the debrief engine " @@ -1591,7 +1643,12 @@ def run_smoke( clock.enter("assertions") turn_failures, turn_warnings = evaluate_turns(results["turns"]) debrief_failures, debrief_warnings = evaluate_debrief( - debrief, rubric_observations_seen=observations_seen + debrief, + rubric_observations_seen=observations_seen, + # A conversation that fell back on every generated turn reaches the + # debrief with nothing to score for a reason that has nothing to do + # with rubric prompting — see UNSCORED_AFTER_FALLBACK_NOTE. + turns_all_fell_back=_all_generated_turns_fell_back(results["turns"]), ) results["warnings"] += turn_warnings + debrief_warnings pipeline_failures = turn_failures + debrief_failures diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index a500f79b..5411eb5a 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -341,6 +341,29 @@ def test_all_generated_turns_falling_back_fails(self) -> None: failures, _ = smoke.evaluate_turns(turns) assert any("fell back" in f for f in failures) + @pytest.mark.parametrize( + "turns, expected", + [ + ([], False), + ([{"label": "npc_opening", "turn_number": 0, "model_generated": False, + "npc_excerpt": "Thanks for coming in."}], False), + ([_turn(used_fallback=True), _turn(turn_number=2, used_fallback=True)], True), + ([_turn(used_fallback=True), _turn(turn_number=2)], False), + # Flags unavailable: an absent used_fallback is not a fallback. + ([_turn(), _turn(turn_number=2)], False), + ], + ) + def test_the_two_verdicts_agree_on_what_all_fell_back_means( + self, turns: list, expected: bool + ) -> None: + # evaluate_turns fails the run for it and evaluate_debrief uses it to + # say the unscored debrief is downstream of it. If the two disagreed, + # one verdict would carry both notes, each naming the other as the + # wrong place to start. + assert smoke._all_generated_turns_fell_back(turns) is expected + failures, _ = smoke.evaluate_turns(list(turns)) + assert any("fell back to the canned" in f for f in failures) is expected + def test_one_fallback_among_several_only_warns(self) -> None: turns = [_turn(used_fallback=True), _turn(turn_number=2), _turn(turn_number=3)] failures, warnings = smoke.evaluate_turns(turns) @@ -566,6 +589,33 @@ def test_no_observations_anywhere_is_named_and_still_worth_chasing(self) -> None assert "not the normal outcome" in no_scores assert "before writing this off" in no_scores + def test_a_zero_caused_by_fallbacks_is_not_blamed_on_rubric_prompting(self) -> None: + # A turn that fell back carries the canned safe utterance and an empty + # rubric_observations list whatever the prompt asked for, so this zero + # says nothing about rubric coverage. UNSCORED_DEBRIEF_NOTE would send + # triage after "the OUTPUT_SCHEMA layer, the starter model pin, + # sampling" while the fallback failure in the same verdict points + # somewhere else entirely. + failures, _ = smoke.evaluate_debrief( + _debrief(scores={}), rubric_observations_seen=0, turns_all_fell_back=True + ) + no_scores = next(f for f in failures if "no rubric dimension scores" in f) + assert smoke.UNSCORED_AFTER_FALLBACK_NOTE in no_scores + assert smoke.UNSCORED_DEBRIEF_NOTE not in no_scores + assert smoke.UNSCORED_WITH_OBSERVATIONS_NOTE not in no_scores + assert "consequence of the fallback failure above" in no_scores + + def test_observations_despite_fallbacks_still_blame_the_debrief_engine(self) -> None: + # Some turns fell back but the rest carried observations: the debrief + # still had something to accumulate, so the fallbacks are not the + # explanation and the regression note is the right one. + failures, _ = smoke.evaluate_debrief( + _debrief(scores={}), rubric_observations_seen=2, turns_all_fell_back=True + ) + no_scores = next(f for f in failures if "no rubric dimension scores" in f) + assert smoke.UNSCORED_WITH_OBSERVATIONS_NOTE in no_scores + assert smoke.UNSCORED_AFTER_FALLBACK_NOTE not in no_scores + def test_an_unknown_observation_count_claims_neither_cause(self) -> None: # The count is what tells the two causes apart, so without it the # failure has to quote both. It must not state one as fact: a message @@ -1751,6 +1801,44 @@ def test_unscored_debrief_is_a_pipeline_failure( assert results["rubric_observations_seen"] == 0 assert any(smoke.UNSCORED_DEBRIEF_NOTE in f for f in results["failures"]) + def test_a_conversation_of_fallbacks_does_not_blame_the_rubric_prompt( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + """The failure mode a 4 B model on CPU actually produces. + + Every turn's output fails validation, so every turn returns the canned + safe utterance with an empty rubric_observations list and the debrief + scores nothing. Both failures land in one verdict, and the unscored one + must not send triage after the OUTPUT_SCHEMA layer, the model pin and + sampling settings when the fallback failure beside it already names the + thing to read. + """ + models_dir, model_id, digest = staged_model + monkeypatch.setattr( + smoke, "_request_json", + _fake_core( + _debrief(scores={}, overall_score=None), + debug_turns=[ + {"turn_number": n, "used_fallback": True, + "used_native_structured_output": False} + for n in range(1, len(smoke.SCRIPTED_PLAYER_TURNS) + 1) + ], + ), + ) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["rubric_observations_seen"] == 0 + assert any("fell back to the canned" in f for f in results["failures"]) + no_scores = next(f for f in results["failures"] if "rubric dimension scores" in f) + assert smoke.UNSCORED_AFTER_FALLBACK_NOTE in no_scores + assert smoke.UNSCORED_DEBRIEF_NOTE not in no_scores + def test_unscored_debrief_with_scorable_turns_is_named_a_regression( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: From 6222ce3bbe619f15e347087a4f0c623f26741743 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 00:18:44 -0400 Subject: [PATCH 37/58] Stop the pipeline remedy from contradicting the doc it mirrors The failure table in docs/real-model-smoke.md carves the unscored debrief out of `pipeline`'s advice -- "Inspect the per-turn used_fallback and replayed_opening flags ... except for an unscored debrief, see Unscored debrief" -- because three of that failure's four causes have nothing to do with the per-turn flags: the debrief engine dropping observations it was handed, a model that volunteered none, and a total the run could not read all start somewhere else entirely. The remedy printed under the banner and in the step summary carried no such carve-out, and that is the copy triage actually reads: the doc is one more click away, and the point of the classification is that the advice next to the verdict is actionable. So a red nightly whose debrief scored nothing sent the reader to the flags, which in three cases out of four say nothing, while the attributed cause sat two lines above in the failure text. All pipeline assertions are raised together under one SmokeFailure, so the remedy cannot be swapped per failure without unpicking that; name the exception in the class remedy instead, the same way the doc does, and point at the doc section. Pinned by a test so the two cannot drift apart again. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 5 ++++- tests/scripts/test_nightly_model_smoke.py | 12 ++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 7dbe7505..f0a647a5 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -334,7 +334,10 @@ class FailureClass: "Both servers were healthy but an end-to-end assertion failed: the model " "ran and produced output the product rejects. Inspect the per-turn " "used_fallback, native-structured-output and replayed_opening flags in " - "the report artifact." + "the report artifact — except for an unscored debrief, where the flags " + "are not where to start: the failure text above has already attributed " + "it, so read that and follow it. See the 'Unscored debrief' section of " + "docs/real-model-smoke.md." ), FailureClass.TIMEOUT: ( "The run exhausted its wall-clock budget in the phase named above. " diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 5411eb5a..3d85b256 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -82,6 +82,18 @@ def test_a_failure_can_override_its_class_remedy(self) -> None: assert exc.exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] assert exc.remedy == "Do this instead." + def test_the_pipeline_remedy_carves_out_the_unscored_debrief(self) -> None: + # All pipeline assertions are raised together, so this one remedy is + # printed under every one of them and cannot be swapped per failure. + # Three of the four unscored-debrief causes are nothing to do with the + # per-turn flags it otherwise sends the reader to -- the docs failure + # table says so, and the advice beside the banner is what triage + # actually reads, so it has to say so too. + remedy = smoke.REMEDIES[smoke.FailureClass.PIPELINE] + assert "unscored debrief" in remedy.lower() + doc = (REPO_ROOT / "docs" / "real-model-smoke.md").read_text(encoding="utf-8") + assert "## Unscored debrief" in doc, "the remedy names a section that must exist" + # --------------------------------------------------------------------------- # Checksum verification From 4de8cdc1fa0b29d297b7be4224dafb51e733b189 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 02:50:15 -0400 Subject: [PATCH 38/58] Stop the unscored-debrief doc from blaming the one cause it exonerates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The closing paragraph of the "Unscored debrief" section referred to its causes by position: "the first is worth a tracking issue for the missing rubric prompt layer". That was true when the section listed two causes and the model-volunteered-nothing one came first. Two later commits inserted causes above it — the unreadable-count case, then the every-generated-turn- fell-back case — and the ordinal was never updated. So the doc now sends a reader who hits the all-fell-back case after the missing rubric prompt layer, which is exactly the investigation the harness goes out of its way to steer them away from: UNSCORED_AFTER_FALLBACK_NOTE ends "Do not start from the rubric prompting; this run says nothing about it", and the bullet two lines above says the debrief "would have had nothing to accumulate even with a rubric layer in place". The remedy and the doc it mirrors contradicted each other again, in the other direction. Name the case instead of its position, and say explicitly that the all-fell-back case is not the product gap, so the next inserted bullet cannot break it. Co-Authored-By: Claude Opus 5 --- docs-site/src/content/docs/dev/real-model-smoke.md | 9 ++++++--- docs/real-model-smoke.md | 9 ++++++--- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index b4fed540..516eb48f 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -208,9 +208,12 @@ So an unscored debrief means one of four quite different things: In every case the run is red: *"scored debrief"* is the acceptance criterion for [#457](https://github.com/outrightmental/ConversationSimulator/issues/457). -No cause is a reason to weaken the check — the first is worth a tracking -issue for the missing rubric prompt layer *and* an investigation of the run that -hit it. +No cause is a reason to weaken the check. The one that is *also* a standing +product gap is **zero observations with turns that did not fall back**: that one +is worth a tracking issue for the missing rubric prompt layer *and* an +investigation of the run that hit it. The all-fell-back case is not — it says +nothing about rubric prompting, and the harness says so rather than sending you +after the prompt layer. --- diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 0e653116..8d649b40 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -204,9 +204,12 @@ So an unscored debrief means one of four quite different things: In every case the run is red: *"scored debrief"* is the acceptance criterion for [#457](https://github.com/outrightmental/ConversationSimulator/issues/457). -No cause is a reason to weaken the check — the first is worth a tracking -issue for the missing rubric prompt layer *and* an investigation of the run that -hit it. +No cause is a reason to weaken the check. The one that is *also* a standing +product gap is **zero observations with turns that did not fall back**: that one +is worth a tracking issue for the missing rubric prompt layer *and* an +investigation of the run that hit it. The all-fell-back case is not — it says +nothing about rubric prompting, and the harness says so rather than sending you +after the prompt layer. --- From f7ea4ccdb17e9f664754b0ac402a59943754e3f0 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 03:15:12 -0400 Subject: [PATCH 39/58] Stop an un-pinned registry entry from reading as corrupt or truncated weights MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `download.url` and `download.sha256` may both hold the literal "PENDING" — model-registry.schema.json documents it as the placeholder for an entry that has been added but not yet pinned — and `resolve_registry_model` accepted it as a value because it only checked for absence. The no-PENDING guard in scripts/validate-registry.py would catch it, but that script runs only in registry-nightly.yml, never per-PR, so an un-pinned `role: starter` entry can reach main and the smoke is the next thing to meet it. Carried into the run, each field produces a confidently wrong verdict: * sha256 "PENDING" builds the cache key `model-gguf-v1-PENDING`, downloads 2.5 GB, and then fails `checksum` (exit 3) — whose remedy says the pinned upstream file was replaced and "do NOT relax this check" — while deleting the file it just fetched. Triage goes looking for drifted upstream bytes that are not the problem. * url "PENDING" raises ValueError out of urllib's Request constructor, which sits outside `_download_with_progress`'s handler, so it lands in main's catch-all as "most likely a bug in the smoke harness" (exit 5). It is neither. An un-pinned entry is exactly the "registry does not describe a usable model" case REGISTRY_REMEDY was written for: a repository problem no re-run can fix. Rejecting it at the single registry lookup means the `--print-registry-model starter` step fails first, before a cache key is built or a byte is fetched. Tests cover all three shapes (url, sha256, both), assert the class, exit code and remedy, and pin the flip side — that the registry as it actually stands is pinned, so the new guard cannot red the nightly on it. The failure-class section of both copies of the doc names the case alongside the other registry problems that report exit 5. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 17 +++---- docs/real-model-smoke.md | 17 +++---- scripts/nightly-model-smoke.py | 38 ++++++++++++++++ tests/scripts/test_nightly_model_smoke.py | 44 +++++++++++++++++++ 4 files changed, 100 insertions(+), 16 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 516eb48f..957cbafb 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -107,14 +107,15 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. -Anything the harness could not carry out — a `registry.yaml` that is malformed -or names no usable `starter` model, a bad command line, a disk that fills up -mid-run, a bug in the harness itself — is also reported as exit 5, with a -message and a remedy that say so, so that **exit 1 only ever means a latency -regression**. Nothing exits with an unclassified traceback, and nothing borrows -a class's exit code without correcting the advice printed beneath it: a broken -registry entry is *not* exit 2, because `download`'s remedy ends "re-run the -job" and no re-run will repair a file in the repository. +Anything the harness could not carry out — a `registry.yaml` that is malformed, +names no usable `starter` model, or still carries the schema's `PENDING` +placeholder instead of a pinned URL and checksum; a bad command line; a disk +that fills up mid-run; a bug in the harness itself — is also reported as exit 5, +with a message and a remedy that say so, so that **exit 1 only ever means a +latency regression**. Nothing exits with an unclassified traceback, and nothing +borrows a class's exit code without correcting the advice printed beneath it: a +broken registry entry is *not* exit 2, because `download`'s remedy ends "re-run +the job" and no re-run will repair a file in the repository. When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 8d649b40..c1037226 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -103,14 +103,15 @@ The distinction that matters most in practice is **2/3 vs 4 vs 5**: a download o checksum failure says nothing about the app, a runtime failure is a crash, and a pipeline failure means the model ran and produced output the product rejected. -Anything the harness could not carry out — a `registry.yaml` that is malformed -or names no usable `starter` model, a bad command line, a disk that fills up -mid-run, a bug in the harness itself — is also reported as exit 5, with a -message and a remedy that say so, so that **exit 1 only ever means a latency -regression**. Nothing exits with an unclassified traceback, and nothing borrows -a class's exit code without correcting the advice printed beneath it: a broken -registry entry is *not* exit 2, because `download`'s remedy ends "re-run the -job" and no re-run will repair a file in the repository. +Anything the harness could not carry out — a `registry.yaml` that is malformed, +names no usable `starter` model, or still carries the schema's `PENDING` +placeholder instead of a pinned URL and checksum; a bad command line; a disk +that fills up mid-run; a bug in the harness itself — is also reported as exit 5, +with a message and a remedy that say so, so that **exit 1 only ever means a +latency regression**. Nothing exits with an unclassified traceback, and nothing +borrows a class's exit code without correcting the advice printed beneath it: a +broken registry entry is *not* exit 2, because `download`'s remedy ends "re-run +the job" and no re-run will repair a file in the repository. When more than one class could apply, the harness reports the strongest evidence rather than the symptom the client happened to see: diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index f0a647a5..90d8b48b 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -464,6 +464,30 @@ def cap(self, timeout_s: float) -> float: "scripts/validate-registry.py) and merge." ) +# The placeholder the registry schema allows in `download.url` and +# `download.sha256` for an entry that has been added but not yet pinned +# (model-registry.schema.json documents both: "or 'PENDING' for pre-release +# entries"). `scripts/validate-registry.py` fails on it — but only in +# registry-nightly.yml, which does not run per-PR, so a `role: starter` entry +# can reach main un-pinned. +# +# It has to be rejected here rather than passed along as a value, because both +# fields carry it into a verdict that blames the wrong thing: +# +# * sha256 "PENDING" makes the cache key `…-PENDING`, downloads 2.5 GB, and +# then fails `checksum` (exit 3) — whose remedy says the pinned upstream +# file was replaced and "do NOT relax this check" — while deleting the +# file it just fetched; +# * url "PENDING" raises out of urllib's Request constructor, which sits +# outside _download_with_progress's handler, so it lands in main's +# catch-all as "most likely a bug in the smoke harness" (exit 5). +# +# An un-pinned entry is neither. It is precisely the "registry does not +# describe a usable model" case REGISTRY_REMEDY exists for, and catching it at +# the one lookup means the `--print-registry-model` step fails first, before +# any cache key is built or any byte is fetched. +REGISTRY_PENDING = "PENDING" + def resolve_registry_model(role: str, registry_path: Path = REGISTRY_PATH) -> Dict[str, Any]: """Return ``{id, url, sha256, size_gb}`` for the registry model with ``role``. @@ -498,6 +522,20 @@ def resolve_registry_model(role: str, registry_path: Path = REGISTRY_PATH) -> Di f"Registry model {model.get('id')!r} is missing download.url or download.sha256", remedy=REGISTRY_REMEDY, ) + unpinned = [ + field + for field, value in (("url", url), ("sha256", sha256)) + if value == REGISTRY_PENDING + ] + if unpinned: + raise SmokeFailure( + FailureClass.PIPELINE, + f"Registry model {model.get('id')!r} is not pinned yet: " + f"{', '.join('download.' + f for f in unpinned)} " + f"still reads {REGISTRY_PENDING!r}. There is nothing to download or " + "verify against, so the smoke cannot run on this model.", + remedy=REGISTRY_REMEDY, + ) return { "id": model["id"], "url": url, diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 3d85b256..8af4bc6c 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -305,6 +305,50 @@ def test_model_without_checksum_is_rejected(self, tmp_path: Path) -> None: assert "sha256" in str(exc_info.value) assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + @pytest.mark.parametrize( + ("url", "sha256", "expected_field"), + [ + ("PENDING", "ab" * 32, "download.url"), + ("https://example.invalid/x.gguf", "PENDING", "download.sha256"), + ("PENDING", "PENDING", "download.url"), + ], + ) + def test_an_unpinned_entry_is_a_registry_problem_not_a_download_one( + self, tmp_path: Path, url: str, sha256: str, expected_field: str + ) -> None: + # "PENDING" is the placeholder model-registry.schema.json allows for an + # entry that has been added but not pinned, and the no-PENDING check in + # scripts/validate-registry.py only runs in registry-nightly.yml — not + # per-PR — so an un-pinned `role: starter` can reach main. Passed along + # as a value it produces two confidently wrong verdicts: a "PENDING" + # sha256 downloads 2.5 GB and then fails `checksum` with "the pinned + # upstream file was replaced … do NOT relax this check", and a "PENDING" + # url raises out of urllib's Request constructor into main's catch-all + # as a suspected harness bug. It is neither: it is the registry failing + # to describe a usable model, which must stop the run at the lookup. + registry = tmp_path / "registry.yaml" + registry.write_text( + "models:\n" + " - id: next-starter\n" + " role: starter\n" + f" download: {{url: {url!r}, sha256: {sha256!r}}}\n", + encoding="utf-8", + ) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.resolve_registry_model("starter", registry) + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + assert exc_info.value.exit_code == 5 + assert expected_field in str(exc_info.value) + assert "not pinned" in str(exc_info.value) + assert exc_info.value.remedy == smoke.REGISTRY_REMEDY + + def test_the_real_starter_entry_is_pinned(self) -> None: + # The flip side: the guard above must not be able to red the nightly on + # the registry as it actually stands. + model = smoke.resolve_registry_model("starter") + assert model["url"] != smoke.REGISTRY_PENDING + assert model["sha256"] != smoke.REGISTRY_PENDING + def test_github_output_is_appended(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: out = tmp_path / "gh-output" monkeypatch.setenv("GITHUB_OUTPUT", str(out)) From 2215e3de746e46675f24290b7c3946e691b4f3a4 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 03:15:47 -0400 Subject: [PATCH 40/58] Stop a registry verdict from naming a model it never resolved MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `render_step_summary` printed `- **Model:** \`{model_id}\`` unconditionally, so a failure in `--print-registry-model` — the one mode that runs before any `--model-id` exists, and the mode that reports a registry naming no usable `starter` entry or not having pinned one yet — led with "**Model:** `None`". That is a field the run never resolved, presented as the model it ran on, in the first line a triager reads above the verdict. Same defect and same fix as the wall clock immediately below it, which already skips rather than defaulting to a measurement the run never took. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 8 +++++++- tests/scripts/test_nightly_model_smoke.py | 18 ++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 90d8b48b..92210104 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1212,8 +1212,14 @@ def render_step_summary(results: Dict[str, Any]) -> str: lines = [ f"## {icon} Real-model smoke: {verdict.upper()}", "", - f"- **Model:** `{results.get('model_id')}`", ] + # Same reasoning as the wall clock below: `--print-registry-model` runs + # before any --model-id exists, and it is the mode that fails when the + # registry names no usable `starter` entry or has not pinned one yet. Its + # summary led with "**Model:** `None`" — a field the run never resolved, + # presented as the model it ran on, directly above the verdict. + if results.get("model_id"): + lines.append(f"- **Model:** `{results['model_id']}`") # Only the smoke run keeps a clock. The pre-run modes (registry lookup, # download, verify) write a summary too, and defaulting their missing # timings to zero printed "Wall clock: 0 s (budget 0 s)" at the top of a diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 8af4bc6c..9fb4387b 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -1148,6 +1148,24 @@ def test_a_pre_run_failure_does_not_claim_a_zero_second_run(self) -> None: assert "Wall clock" not in text assert "`download`" in text assert smoke.REMEDIES[smoke.FailureClass.DOWNLOAD] in text + assert "**Model:** `qwen3-4b`" in text + + def test_a_registry_failure_does_not_name_a_model_it_never_resolved(self) -> None: + # --print-registry-model runs before any --model-id exists, and it is + # the mode that fails on a registry with no usable (or not yet pinned) + # `starter` entry. Rendering `model_id` unconditionally put + # "**Model:** `None`" at the very top of that verdict. + text = smoke.render_step_summary({ + "verdict": "fail", + "model_id": None, + "failure_class": smoke.FailureClass.PIPELINE, + "exit_code": 5, + "failures": ["No model with role 'starter' in model-registry/registry.yaml"], + "remedy": smoke.REGISTRY_REMEDY, + }) + assert "Model:" not in text + assert "None" not in text + assert smoke.REGISTRY_REMEDY in text def test_summary_is_not_written_without_the_github_env_var( self, monkeypatch: pytest.MonkeyPatch From 1591b530711dec8a05dc829a496cf5c893d12309 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 03:17:18 -0400 Subject: [PATCH 41/58] Stop an unrequestable URL from reading as a bug in the harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit urllib raises ValueError out of Request's constructor — not out of urlopen — for a string it cannot read as a URL ("unknown url type"), and that call sat above `_download_with_progress`'s handler. So the failure escaped every classification in the file and landed in main's catch-all, which reported "most likely a bug in the smoke harness" and advised re-running the harness locally with the same arguments. The argument is the problem; re-running it reproduces it. That is reachable from the harness's own documentation: the local repro block in docs/real-model-smoke.md hands the reader `--model-url ""` to substitute, and pasting it verbatim lands exactly here. A registry entry whose `download.url` lost its scheme does the same. Classified `pipeline` rather than `download`: nothing was fetched, so neither "check that Hugging Face is reachable" nor "re-run the job" applies, and `pipeline` is already the class this harness uses for inputs it cannot act on. It carries its own remedy naming both places the string can come from, since the class default ("inspect the per-turn used_fallback flags") is meaningless for a run that never played a turn. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 31 ++++++++++++++++++++++- tests/scripts/test_nightly_model_smoke.py | 26 +++++++++++++++++++ 2 files changed, 56 insertions(+), 1 deletion(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 92210104..3dde36d0 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -561,7 +561,36 @@ def _emit_github_output(values: Dict[str, Any]) -> None: def _download_with_progress(url: str, dest: Path) -> None: dest.parent.mkdir(parents=True, exist_ok=True) print(f" Downloading {url[:80]}…") - req = urllib.request.Request(url, headers={"User-Agent": "convsim-smoke/1.0"}) + try: + req = urllib.request.Request(url, headers={"User-Agent": "convsim-smoke/1.0"}) + except ValueError as exc: + # urllib raises out of the *constructor* for a string it cannot read as + # a URL ("unknown url type"), i.e. before the handler below exists. That + # escaped to main's catch-all and was reported as "most likely a bug in + # the smoke harness", with a remedy telling the reader to re-run with + # the same arguments — when the argument is the whole problem. + # + # Not `download` either: nothing was fetched and no re-run or registry + # URL check changes the outcome. This is the harness being handed an + # input it cannot act on, which is what `pipeline` covers for everything + # else of that kind, with a remedy naming both places the string can + # come from. The realistic source is the local repro in + # docs/real-model-smoke.md, whose --model-url is a placeholder the + # reader is meant to substitute. + raise SmokeFailure( + FailureClass.PIPELINE, + f"Not a usable download URL: {url[:120]!r} ({exc})", + remedy=( + "Nothing was fetched — the string is not a URL urllib can " + "request. If it came from --model-url on the command line, " + "substitute the real URL printed by " + "`--print-registry-model starter` (the repro block in " + "docs/real-model-smoke.md shows a placeholder, not a value). " + "If it came from the registry, repair download.url in " + "model-registry/registry.yaml and validate with " + "scripts/validate-registry.py --url-check." + ), + ) from exc try: with urllib.request.urlopen(req, timeout=600) as resp, open(dest, "wb") as f: total = int(resp.headers.get("Content-Length", 0)) or None diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 9fb4387b..7de0dcb3 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -223,6 +223,32 @@ def test_a_broken_transfer_leaves_no_partial_file_to_checksum_fail_on( smoke._download_with_progress("https://example.invalid/m.gguf", dest) assert not dest.exists() + @pytest.mark.parametrize( + "url", + [ + "", # the repro block's placeholder, pasted verbatim + "example.invalid/m.gguf", # scheme omitted + ], + ) + def test_an_unrequestable_url_is_not_reported_as_a_harness_bug( + self, url: str, tmp_path: Path + ) -> None: + # urllib raises ValueError out of Request's *constructor* for a string + # it cannot read as a URL, which used to sit above the handler — so it + # escaped to main's catch-all as "most likely a bug in the smoke + # harness", advising a re-run with the same arguments. The arguments are + # the problem, and the local repro in docs/real-model-smoke.md hands the + # reader a --model-url placeholder to substitute. + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._download_with_progress(url, tmp_path / "m.gguf") + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + assert "not a usable download url" in str(exc_info.value).lower() + # Its own remedy, not the class default: "inspect the per-turn + # used_fallback flags" is useless when no turn was ever played. + assert exc_info.value.remedy != smoke.REMEDIES[smoke.FailureClass.PIPELINE] + assert "--model-url" in exc_info.value.remedy + assert "registry" in exc_info.value.remedy + # --------------------------------------------------------------------------- # Registry resolution From c00b4539f69b9dd3fb3692424b38c91b95f4e7b9 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 03:19:40 -0400 Subject: [PATCH 42/58] Stop a debrief summary that is not text from reading as a harness bug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit evaluate_debrief guards `scores` and `overall_score` with isinstance, but read `summary` as a string unconditionally — and run_smoke's report assembly excerpted it even earlier, before evaluate_debrief ran at all. A summary that arrived as anything but text therefore raised AttributeError out of `str.split`/`str.strip`, escaped the assertion layer into run_smoke's catch-all, and was reported as "most likely a bug in the smoke harness itself, not in the product" with the harness-bug remedy — the one verdict that is certainly wrong when it is the product's own response contract that changed, and one that sends triage to the traceback instead of to convsim-core. The harness already settles this question the other way everywhere else: _npc_turn_content type-guards `content`, _rubric_observation_count returns None rather than 0 for an unreadable list, and _debug_flags_by_turn drops an unkeyable entry precisely so a malformed payload cannot be reported as a bug in this file. The debrief summary was the one field left out. So check the type before measuring the length, and fail as `pipeline` naming the type and the value that arrived; and route the report assembly through a tolerant _excerpt_any so the earlier read cannot raise before the classified verdict is reached. _excerpt_any renders a non-string with repr, keeping the shape visible in the artifact rather than coercing it to an empty string, and deliberately does not judge: a dict's repr clears MIN_SUMMARY_CHARS easily, so the type check in evaluate_debrief is what fails the run. Both call sites are covered, including an end-to-end run_smoke case that asserts exit 5 with the product remedy and no "bug in the smoke harness" text — guarding the assertion layer alone would have left the raise reachable. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 45 ++++++++++++-- tests/scripts/test_nightly_model_smoke.py | 74 +++++++++++++++++++++++ 2 files changed, 115 insertions(+), 4 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 3dde36d0..13d7a447 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -926,6 +926,26 @@ def _excerpt(text: str) -> str: return collapsed[:EXCERPT_CHARS] + ("…" if len(collapsed) > EXCERPT_CHARS else "") +def _excerpt_any(value: Any) -> str: + """Excerpt a server-supplied value that *should* be a string, whatever it is. + + ``_excerpt`` calls ``str.split``, so handing it a non-string raises + ``AttributeError``. Where that value came out of a convsim-core response — + the debrief summary is the one such field the harness both excerpts into the + report and asserts on — the raise escapes into ``run_smoke``'s catch-all and + reports a changed product response contract as "most likely a bug in the + smoke harness", which is the one thing it is certainly not. + + ``repr`` rather than ``str``, so the artifact shows the shape that arrived + instead of a coerced empty string. Telling the run it is *wrong* stays + ``evaluate_debrief``'s job: a dict's repr is easily long enough to satisfy + the summary length check, so nothing here may be mistaken for acceptance. + """ + if value is None: + return "" + return _excerpt(value if isinstance(value, str) else repr(value)) + + def _replays_opening(npc_text: str, opening_text: str) -> bool: """True when an NPC reply opens by reciting the authored opening verbatim. @@ -1135,11 +1155,24 @@ def evaluate_debrief( elif not 0 <= overall <= 100: failures.append(f"Debrief overall_score {overall} is outside [0, 100]") - summary = debrief.get("summary") or "" - if len(summary.strip()) < MIN_SUMMARY_CHARS: + summary = debrief.get("summary") + if summary is not None and not isinstance(summary, str): + # Checked as a type before it is measured as a length, for the same + # reason _rubric_observation_count returns None rather than 0 for an + # unreadable list: len(summary.strip()) raises AttributeError here, and + # an AttributeError out of the assertion layer lands in run_smoke's + # catch-all as "most likely a bug in the smoke harness" — the one + # verdict that is certainly wrong when it is the product's own response + # contract that changed. A debrief whose summary is not text is a + # `pipeline` failure like any other output the product rejects. + failures.append( + f"Debrief summary is not a string (got {type(summary).__name__}): " + f"{_excerpt_any(summary)}" + ) + elif len((summary or "").strip()) < MIN_SUMMARY_CHARS: failures.append( f"Debrief summary is shorter than {MIN_SUMMARY_CHARS} characters " - f"({len(summary.strip())} chars)" + f"({len((summary or '').strip())} chars)" ) if debrief.get("used_fallback"): @@ -1693,7 +1726,11 @@ def run_smoke( "total_turns": debrief.get("total_turns"), "scores": debrief.get("scores"), "overall_score": debrief.get("overall_score"), - "summary_excerpt": _excerpt(debrief.get("summary") or ""), + # _excerpt_any, not _excerpt: this report is assembled *before* + # evaluate_debrief runs, so a summary that is not text would raise + # here and the classified verdict evaluate_debrief was about to + # reach would never be assembled. See _excerpt_any. + "summary_excerpt": _excerpt_any(debrief.get("summary")), "strength_count": len(debrief.get("strengths") or []), "improvement_count": len(debrief.get("improvements") or []), "turning_point_count": len(debrief.get("turning_points") or []), diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 7de0dcb3..747f8f7b 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -748,6 +748,35 @@ def test_empty_summary_fails(self) -> None: failures, _ = smoke.evaluate_debrief(_debrief(summary=" ")) assert any("summary is shorter" in f for f in failures) + def test_an_absent_summary_is_still_measured_as_too_short(self) -> None: + failures, _ = smoke.evaluate_debrief(_debrief(summary=None)) + assert any("summary is shorter" in f for f in failures) + + @pytest.mark.parametrize( + "summary", + [ + # Long enough that a repr would clear MIN_SUMMARY_CHARS, so the type + # check cannot be mistaken for the length check passing. + {"text": "You gave concrete examples but hedged on the trade-off."}, + ["You gave concrete examples but hedged on the trade-off."], + 42, + ], + ) + def test_a_summary_that_is_not_text_is_a_product_failure_not_a_harness_bug( + self, summary: object + ) -> None: + # len(summary.strip()) raises AttributeError on all of these, and an + # AttributeError out of the assertion layer reaches run_smoke's catch-all, + # which reports "most likely a bug in the smoke harness" -- the one + # verdict that is certainly wrong for a changed response contract. + failures, _ = smoke.evaluate_debrief(_debrief(summary=summary)) + assert any("summary is not a string" in f for f in failures) + # Named as a type, and the value shown, so triage does not have to go and + # fetch the artifact to find out what arrived. + assert any(type(summary).__name__ in f for f in failures) + # And never reported as a length problem: nothing here is about brevity. + assert not any("summary is shorter" in f for f in failures) + def test_fallback_narrative_only_warns_because_scores_are_still_real(self) -> None: failures, warnings = smoke.evaluate_debrief(_debrief(used_fallback=True)) assert failures == [] @@ -1305,6 +1334,18 @@ def test_excerpt_is_bounded_and_collapsed(self) -> None: assert len(excerpt) <= smoke.EXCERPT_CHARS + 1 # + the ellipsis assert excerpt.endswith("…") + def test_excerpting_a_non_string_shows_its_shape_instead_of_raising(self) -> None: + # _excerpt alone raises AttributeError here (str.split), which is why the + # report assembly in run_smoke goes through _excerpt_any. + assert smoke._excerpt_any({"text": "hi"}) == "{'text': 'hi'}" + assert smoke._excerpt_any(42) == "42" + + def test_excerpting_an_absent_value_is_empty_not_the_word_none(self) -> None: + assert smoke._excerpt_any(None) == "" + + def test_excerpting_a_string_is_unchanged_by_the_tolerant_wrapper(self) -> None: + assert smoke._excerpt_any("a\n\n b") == smoke._excerpt("a\n\n b") + # --------------------------------------------------------------------------- # Scripted conversation shape @@ -1901,6 +1942,39 @@ def test_unscored_debrief_is_a_pipeline_failure( assert results["rubric_observations_seen"] == 0 assert any(smoke.UNSCORED_DEBRIEF_NOTE in f for f in results["failures"]) + def test_a_debrief_summary_that_is_not_text_is_a_product_verdict( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + """A changed response contract must not be reported as a harness bug. + + The summary is read twice on the way to a verdict, and the *first* read is + the report assembly in run_smoke, which runs before evaluate_debrief: a + guard in the assertion layer alone leaves the raise reachable. Both go + through the tolerant excerpt, so the run reaches its classified verdict + and the artifact still shows what arrived. + """ + models_dir, model_id, digest = staged_model + summary = {"text": "You gave concrete examples but hedged on the trade-off."} + monkeypatch.setattr( + smoke, "_request_json", _fake_core(_debrief(summary=summary)) + ) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failure_class"] == smoke.FailureClass.PIPELINE + assert results["failed_phase"] == "assertions" + assert any("summary is not a string" in f for f in results["failures"]) + # The product's remedy, not the harness-bug one the catch-all prints. + assert results["remedy"] == smoke.REMEDIES[smoke.FailureClass.PIPELINE] + assert not any("bug in the smoke harness" in f for f in results["failures"]) + # And the payload that caused it survives into the artifact. + assert "hedged on the trade-off" in results["debrief"]["summary_excerpt"] + def test_a_conversation_of_fallbacks_does_not_blame_the_rubric_prompt( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: From 924a6032c6b0f2a45c1fbaf9a0b3fed9af2e682c Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 03:20:38 -0400 Subject: [PATCH 43/58] Refuse an unreadable runtime report instead of blaming the harness for it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last read of a convsim-core response that could raise instead of producing a verdict: `health.get("llm_runtime", {}).get("runtime_id")`. A missing key was already handled — the `{}` default falls through to the clean "core reports None" pipeline refusal — but an explicit JSON null is a *present* key, so the default never applies and .get() raises AttributeError on None. That escapes to run_smoke's catch-all and reports a changed /health contract as "most likely a bug in the smoke harness", exactly as the debrief summary did before c00b453. `or {}` instead, so both shapes land on the refusal that already exists. This check is the one standing between the nightly and a green run on the fake runtime, so it has to produce a verdict for every answer /health can give, not just the well-formed ones. Covered for both shapes, asserting the refusal names runtime_id and that the harness does not accuse itself. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 6 ++++- tests/scripts/test_nightly_model_smoke.py | 30 +++++++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 13d7a447..834cdff1 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1579,7 +1579,11 @@ def run_smoke( # Sanity-check that the real runtime is wired (not the fake default). health = _request_json(f"{base}/health", timeout=clock.cap(30.0)) - runtime_id = health.get("llm_runtime", {}).get("runtime_id") + # `or {}`, not a `{}` default: an absent llm_runtime already falls + # through to the clean "core reports None" verdict below, but an explicit + # JSON null would make .get() raise AttributeError and report a changed + # /health contract as a bug in this harness. Same reason as _excerpt_any. + runtime_id = (health.get("llm_runtime") or {}).get("runtime_id") results["runtime_id"] = runtime_id print(f"[smoke] convsim-core ready (runtime_id={runtime_id}).") if runtime_id != "llama_cpp": diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 747f8f7b..9e9cdaf8 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -2240,6 +2240,36 @@ def _fake_runtime(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> results = json.loads(report.read_text(encoding="utf-8")) assert "runtime_id" in results["failures"][0] + @pytest.mark.parametrize("health", [{}, {"llm_runtime": None}]) + def test_an_unreadable_runtime_report_is_refused_not_called_a_harness_bug( + self, health: dict, staged_model, fake_servers, + monkeypatch: pytest.MonkeyPatch, tmp_path: Path, + ) -> None: + """An absent *or null* llm_runtime must still reach the refusal verdict. + + The smoke must never play a turn without having confirmed the real + runtime, so an unreadable /health is a refusal like any other -- and a + `{}` default would have let an explicit null raise AttributeError and + report a changed /health contract as a bug in the harness instead. + """ + models_dir, model_id, digest = staged_model + + def _unreadable(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + assert url.endswith("/health"), "must not play a turn on an unknown runtime" + return health + + monkeypatch.setattr(smoke, "_request_json", _unreadable) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + results = json.loads(report.read_text(encoding="utf-8")) + assert "runtime_id" in results["failures"][0] + assert not any("bug in the smoke harness" in f for f in results["failures"]) + def test_checksum_drift_stops_the_run_before_any_server_starts( self, staged_model, monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: From 134fe781b2d2c7299d6c129871d706ee83df7db8 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 03:40:51 -0400 Subject: [PATCH 44/58] Fail a conversation split between fallbacks and recited openings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two "no real turns" checks each fail only when *every* model-generated turn is bad, and warn otherwise: recovering from one rejected turn is the product working, not a reason to red the nightly. But the two anomalies can split a conversation between them — turn 1 recites the authored opening, turns 2 and 3 fall back to the canned safe utterance — and then neither "all of them" condition holds while not one turn is a reply. evaluate_turns returned two warnings and no failure, so the job went green on a conversation that contains no real NPC answer at all: exactly the verdict both checks exist to prevent, reachable because each was written without the other in view. It is not a contrived combination either. The PR's own evidence is that the previous harness logged a recited opening on the first player turn on every one of the last seven nightlies, and a 4 B model does occasionally emit turn output the validator rejects on the others. So the policies now compose: when no generated turn is both real model output and an answer, the run fails, with a message that names both counts and says why neither alone was enough. The two single-cause failures are unchanged and are not double-reported — the composed check is skipped whenever one of them already fired — and a run with even one good turn keeps its two warnings and its green verdict. An absent used_fallback flag still cannot convict anything, so a run without the best-effort debug endpoint cannot reach the new failure. Documented in both copies of docs/real-model-smoke.md. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 7 +++ docs/real-model-smoke.md | 7 +++ scripts/nightly-model-smoke.py | 32 +++++++++++- tests/scripts/test_nightly_model_smoke.py | 52 +++++++++++++++++++ 4 files changed, 97 insertions(+), 1 deletion(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 957cbafb..76ac0f3e 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -67,6 +67,13 @@ went green on a conversation that never happened. The harness therefore records fallbacks: one such turn warns, *every* generated turn doing it fails, because then the model answered nothing and the run proves nothing. +The two policies also compose, because a conversation can be *split* between +them — one turn recites the opening, the others fall back — and trip neither +"all of them" check while still containing no reply at all. Each check warns +rather than fails on its own only because a run that recovers from one bad turn +is the product working; when nothing recovered there is nothing to credit, so a +run with no turn that is both real model output *and* an answer fails as well. + What counts as a recital is a *leading prefix* match, `REPLAY_PREFIX_CHARS` (40 collapsed characters) of the authored opening reproduced at the head of the reply — not the whole opening. Those nightly logs printed only `npc_text[:80]` diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index c1037226..992169f7 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -63,6 +63,13 @@ went green on a conversation that never happened. The harness therefore records fallbacks: one such turn warns, *every* generated turn doing it fails, because then the model answered nothing and the run proves nothing. +The two policies also compose, because a conversation can be *split* between +them — one turn recites the opening, the others fall back — and trip neither +"all of them" check while still containing no reply at all. Each check warns +rather than fails on its own only because a run that recovers from one bad turn +is the product working; when nothing recovered there is nothing to credit, so a +run with no turn that is both real model output *and* an answer fails as well. + What counts as a recital is a *leading prefix* match, `REPLAY_PREFIX_CHARS` (40 collapsed characters) of the authored opening reproduced at the head of the reply — not the whole opening. Those nightly logs printed only `npc_text[:80]` diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 834cdff1..27fd5aec 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1015,6 +1015,11 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: the player (see ``_replays_opening``). Unlike the parse flags this is computed from the response the harness already holds, so it is never missing. + + The two policies also have to compose: a conversation split between them — + one turn recites, the rest fall back — trips neither "all of them" check + while still containing no reply, so a run with no turn that is both real + output *and* an answer fails as well. """ failures: List[str] = [] warnings: List[str] = [] @@ -1043,7 +1048,8 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: ) fallbacks = [t for t in generated if t.get("used_fallback")] - if _all_generated_turns_fell_back(turns): + all_fell_back = _all_generated_turns_fell_back(turns) + if all_fell_back: failures.append( f"All {len(generated)} model-generated NPC turns fell back to the canned " "safe utterance; the real model produced no usable turn output" @@ -1073,6 +1079,30 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: "answering them: real output, but not a reply. Recurring in the same turn " "position every night is a turn-prompt problem, not an unlucky sample" ) + + # Neither check above fires when the generated turns are *split* between the + # two anomalies — one recites the authored opening, the others fall back — + # and that conversation contains no NPC reply at all. Each check warns + # rather than fails on its own because a run that recovers from one bad turn + # is the product working; the union of them leaves nothing that recovered, + # which is the failure both checks exist to catch, so it has to fail too. + # Reachable in practice: the previous harness logged a recited opening on + # the first player turn every night, and a 4 B model does occasionally emit + # output the validator rejects on the rest. + mixed = not all_fell_back and len(echoes) != len(generated) + if mixed and not [ + t + for t in generated + if not t.get("used_fallback") and not t.get("replayed_opening") + ]: + failures.append( + f"Not one of the {len(generated)} model-generated NPC turns was a reply: " + f"{len(fallbacks)} fell back to the canned safe utterance and " + f"{len(echoes)} began by reciting the authored opening line. Neither " + "count alone would fail the run, but between them they account for " + "every generated turn, so no player turn was answered and this run " + "proves nothing about real NPC replies" + ) return (failures, warnings) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 9e9cdaf8..4ac32be3 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -507,6 +507,58 @@ def test_one_turn_reciting_the_opening_only_warns(self) -> None: assert failures == [] assert any("reciting the authored opening" in w for w in warnings) + def test_a_conversation_split_between_the_two_anomalies_still_fails(self) -> None: + # Neither "all fell back" nor "all recited" fires, and yet not one turn + # was a reply. Each check warns on its own because recovering from a bad + # turn is the product working; together they leave nothing recovered. + turns = [ + _turn(replayed_opening=True), + _turn(turn_number=2, used_fallback=True), + _turn(turn_number=3, used_fallback=True), + ] + failures, _ = smoke.evaluate_turns(turns) + assert any("Not one of the 3" in f and "was a reply" in f for f in failures) + # The two single-cause failures stay off: each is false here, and + # printing them would misdescribe the run. + assert not any("All 3" in f for f in failures) + + def test_a_single_good_turn_keeps_a_mixed_conversation_green(self) -> None: + # One real reply among a fallback and a recital is the "the product + # recovered" case both warnings exist for, not a failure. + turns = [ + _turn(replayed_opening=True), + _turn(turn_number=2, used_fallback=True), + _turn(turn_number=3), + ] + failures, warnings = smoke.evaluate_turns(turns) + assert failures == [] + assert len(warnings) == 2 + + def test_the_composed_check_does_not_double_report_all_fallbacks(self) -> None: + turns = [_turn(used_fallback=True), _turn(turn_number=2, used_fallback=True)] + failures, _ = smoke.evaluate_turns(turns) + assert len(failures) == 1 + assert "fell back to the canned" in failures[0] + + def test_the_composed_check_does_not_double_report_all_recitals(self) -> None: + turns = [_turn(replayed_opening=True), _turn(turn_number=2, replayed_opening=True)] + failures, _ = smoke.evaluate_turns(turns) + assert len(failures) == 1 + assert "reciting the authored opening" in failures[0] + + def test_absent_parse_flags_cannot_produce_the_composed_failure(self) -> None: + # Without the debug endpoint no turn is known to have fallen back, so + # the only turns left are judged on replayed_opening alone — the + # composed check must not convict a run it has no evidence about. + turns = [ + {"label": "player_turn_1", "turn_number": 1, "model_generated": True, + "npc_excerpt": "Go on.", "replayed_opening": True}, + {"label": "player_turn_2", "turn_number": 2, "model_generated": True, + "npc_excerpt": "Go on.", "replayed_opening": False}, + ] + failures, _ = smoke.evaluate_turns(turns) + assert not any("was a reply" in f for f in failures) + def test_a_recited_opening_is_not_mistaken_for_a_fallback(self) -> None: # The two have different causes and different fixes, so the verdict # must not describe one as the other. From b36179980a2b696f00feebac03ae2f516eb46b8e Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 04:05:28 -0400 Subject: [PATCH 45/58] Warn that --verify-only deletes the model it finds drifted MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doc's "Useful extras" list and the flag's own --help described --verify-only as re-verifying an already-downloaded model and exiting, which reads as a read-only check. It is not: verify_model_checksum deletes on mismatch by default, so pointing it at a drifted or truncated GGUF destroys a 2.5 GB download. That default is right for the two callers the nightly uses — leaving known-bad bytes on disk would make the next run fail on the same file — but --verify-only exists only for the local repro, where someone reaching for a checker does not expect to have to re-download 2.5 GB afterwards. Say so in the three places a reader meets the flag: the module docstring's usage block, the argparse help, and both copies of docs/real-model-smoke.md. Co-Authored-By: Claude Opus 5 --- docs-site/src/content/docs/dev/real-model-smoke.md | 5 ++++- docs/real-model-smoke.md | 5 ++++- scripts/nightly-model-smoke.py | 7 +++++-- 3 files changed, 13 insertions(+), 4 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 76ac0f3e..8635f92a 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -353,7 +353,10 @@ python scripts/nightly-model-smoke.py \ Useful extras: -- `--verify-only` — re-verify an already-downloaded model and exit. +- `--verify-only` — re-verify an already-downloaded model and exit. Like every + other checksum check here, a mismatch **deletes** the file (exit 3), so a + re-run re-downloads rather than re-verifying the same bad bytes — do not + reach for it as a read-only inspection of a 2.5 GB download you want to keep. - `--models-dir ` — look for `.gguf` somewhere other than `~/.convsim/models/llm/`. - `--wall-clock-budget-s` — the self-imposed deadline (default 1200 s). diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 992169f7..beadad48 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -349,7 +349,10 @@ python scripts/nightly-model-smoke.py \ Useful extras: -- `--verify-only` — re-verify an already-downloaded model and exit. +- `--verify-only` — re-verify an already-downloaded model and exit. Like every + other checksum check here, a mismatch **deletes** the file (exit 3), so a + re-run re-downloads rather than re-verifying the same bad bytes — do not + reach for it as a read-only inspection of a 2.5 GB download you want to keep. - `--models-dir ` — look for `.gguf` somewhere other than `~/.convsim/models/llm/`. - `--wall-clock-budget-s` — the self-imposed deadline (default 1200 s). diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 27fd5aec..bf62f467 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -78,7 +78,8 @@ class exit meaning python scripts/nightly-model-smoke.py --download-only \\ --model-id --model-url --model-sha256 -Verify an already-cached file without running the smoke:: +Verify an already-cached file without running the smoke (a mismatch *deletes* +the file — see ``verify_model_checksum``):: python scripts/nightly-model-smoke.py --verify-only \\ --model-id --model-sha256 @@ -1987,7 +1988,9 @@ def main(argv: Optional[Sequence[str]] = None) -> int: parser.add_argument("--download-only", action="store_true", help="Download and verify the model file, then exit.") parser.add_argument("--verify-only", action="store_true", - help="Verify the on-disk model against --model-sha256, then exit.") + help="Verify the on-disk model against --model-sha256, then exit. " + "A mismatch deletes the file, so a re-run re-downloads it " + "instead of re-verifying the same bad bytes.") parser.add_argument("--model-url", default=None, help="Download URL (required for --download-only).") parser.add_argument("--model-sha256", default=None, From f931bbaa2503ab049666185bfb52c0277ac6b742 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 08:42:21 -0400 Subject: [PATCH 46/58] Stop the unscored-debrief remedy from offering a fix that changes nothing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The note appended to a "no scores" failure told the reader to fix the standing weakness "by giving the turn prompt a rubric layer or playing a scenario that defines one", and the doc said the same. The second half is a dead end: `ScenarioData` — the dataclass the turn composer is handed — has no rubric field at all, and no prompt layer reads one, so the official job-interview-basic pack's `behavioral_interview`, which *does* define a rubric, would compose a byte-identical turn prompt. The full edition also resolves the built-in catalogue before installed packs, so the smoke could not reach that scenario by id even if it helped. So the one cause that is also a product gap now names what fixing it actually takes — a prompt layer *and* a rubric to put in it — and says outright that switching scenarios is not it, rather than sending a triager to change `SCENARIO_ID` and find nothing moved. Both claims are pinned by tests, since they are statements about product source this harness deliberately does not import: a rubric appearing in layers.py or in ScenarioData now fails, with the message naming the note and the doc section to revisit. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 22 +++++++----- docs/real-model-smoke.md | 22 +++++++----- scripts/nightly-model-smoke.py | 23 ++++++++----- tests/scripts/test_nightly_model_smoke.py | 34 +++++++++++++++++++ 4 files changed, 77 insertions(+), 24 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 8635f92a..dec767be 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -157,11 +157,16 @@ which of the causes below applies. The debrief's dimension scores are accumulated entirely from `rubric_observations` that the model volunteers on each NPC turn. Nothing asks -it for them: the built-in `behavioral_interview` scenario defines no rubric, no -prompt layer names any rubric dimensions, and the only hint the model gets is -the bare `rubric_observations` array in the embedded output schema — whose empty -list the schema accepts. The fake runtime always returns `[]`, which is why the -release-time playthrough asserts only that `scores` *is* a dict. +it for them: no prompt layer names any rubric dimensions, and the only hint the +model gets is the bare `rubric_observations` array in the embedded output +schema — whose empty list the schema accepts. Nor could a scenario supply one. +`ScenarioData` (in `convsim_prompt.types`) has no rubric field at all, so the +official `job-interview-basic` pack's `behavioral_interview` — which *does* +define a rubric — would compose the identical turn prompt; and the full edition +resolves the built-in catalogue before installed packs, so this job always plays +the rubric-less built-in scenario regardless. The fake runtime always returns +`[]`, which is why the release-time playthrough asserts only that `scores` *is* +a dict. That thin prompt coverage is the standing weakness that makes an unscored debrief reachable at all. It is *not*, however, the expected outcome: in local @@ -189,9 +194,10 @@ So an unscored debrief means one of four quite different things: but since the starter model does normally answer that hint, first check what changed about what reaches the model: the `OUTPUT_SCHEMA` prompt layer, the registry's `starter` pin, the adapter's JSON-schema constraint, sampling - settings. Then fix the weakness itself in the product (give the turn prompt a - rubric layer, or play a scenario that defines one), not by relaxing the - assertion. + settings. Then fix the weakness itself in the product — the turn prompt needs + a rubric layer *and* a rubric to put in it, since `ScenarioData` carries none + — not by relaxing the assertion, and not by pointing the smoke at a + rubric-defining scenario, which would compose the same prompt. - **`rubric_observations_seen` is above 0 — a real regression.** The turns returned observations and the debrief scored none of them, so the model did its part. The debrief engine does not score the validated observations the diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index beadad48..eca2e8d9 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -153,11 +153,16 @@ which of the causes below applies. The debrief's dimension scores are accumulated entirely from `rubric_observations` that the model volunteers on each NPC turn. Nothing asks -it for them: the built-in `behavioral_interview` scenario defines no rubric, no -prompt layer names any rubric dimensions, and the only hint the model gets is -the bare `rubric_observations` array in the embedded output schema — whose empty -list the schema accepts. The fake runtime always returns `[]`, which is why the -release-time playthrough asserts only that `scores` *is* a dict. +it for them: no prompt layer names any rubric dimensions, and the only hint the +model gets is the bare `rubric_observations` array in the embedded output +schema — whose empty list the schema accepts. Nor could a scenario supply one. +`ScenarioData` (in `convsim_prompt.types`) has no rubric field at all, so the +official `job-interview-basic` pack's `behavioral_interview` — which *does* +define a rubric — would compose the identical turn prompt; and the full edition +resolves the built-in catalogue before installed packs, so this job always plays +the rubric-less built-in scenario regardless. The fake runtime always returns +`[]`, which is why the release-time playthrough asserts only that `scores` *is* +a dict. That thin prompt coverage is the standing weakness that makes an unscored debrief reachable at all. It is *not*, however, the expected outcome: in local @@ -185,9 +190,10 @@ So an unscored debrief means one of four quite different things: but since the starter model does normally answer that hint, first check what changed about what reaches the model: the `OUTPUT_SCHEMA` prompt layer, the registry's `starter` pin, the adapter's JSON-schema constraint, sampling - settings. Then fix the weakness itself in the product (give the turn prompt a - rubric layer, or play a scenario that defines one), not by relaxing the - assertion. + settings. Then fix the weakness itself in the product — the turn prompt needs + a rubric layer *and* a rubric to put in it, since `ScenarioData` carries none + — not by relaxing the assertion, and not by pointing the smoke at a + rubric-defining scenario, which would compose the same prompt. - **`rubric_observations_seen` is above 0 — a real regression.** The turns returned observations and the debrief scored none of them, so the model did its part. The debrief engine does not score the validated observations the diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index bf62f467..2708e71e 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -192,13 +192,18 @@ class exit meaning # Appended to the "no scores" failure. The debrief's dimension scores are # accumulated purely from rubric_observations the *model* volunteers, and -# nothing asks it for them: the built-in behavioral_interview scenario defines -# no rubric, and no prompt layer names one (see +# nothing asks it for them: no prompt layer names a rubric dimension (see # packages/prompt-composer/.../layers.py — the model's only hint is the bare # rubric_observations array in the embedded output schema, whose empty list the -# schema accepts). So `pipeline`'s stock advice — inspect the per-turn -# used_fallback flags — would send triage looking for a parse failure that did -# not happen. +# schema accepts). Nor is there any route by which a scenario could supply +# one: convsim_prompt.types.ScenarioData carries no rubric field at all, so the +# official job-interview-basic pack's behavioral_interview — which *does* +# define a rubric — would compose the identical turn prompt, and the full +# edition resolves the catalogue first anyway, so this job always plays the +# rubric-less built-in. Switching scenarios is therefore not a fix, and the +# remedy must not offer it as one. So `pipeline`'s stock advice — inspect the +# per-turn used_fallback flags — would send triage looking for a parse failure +# that did not happen. # # That unguided array is NOT normally empty, though, which is why this note # does not simply write the failure off: in local verification against the real @@ -226,9 +231,11 @@ class exit meaning "starter model volunteered observations on every scripted turn in local " "verification. So check what changed about what reaches the model (the " "OUTPUT_SCHEMA layer, the starter model pin, sampling) before writing this " - "off, and fix the weakness itself by giving the turn prompt a rubric layer " - "or playing a scenario that defines one. See the 'Unscored debrief' section " - "of docs/real-model-smoke.md." + "off, and fix the weakness itself in the product: the turn prompt needs a " + "rubric layer AND a rubric to put in it (ScenarioData carries no rubric " + "field, so pointing the smoke at a scenario that defines one composes the " + "same prompt). See the 'Unscored debrief' section of " + "docs/real-model-smoke.md." ) # The other cause of an unscored debrief, and the one that *is* a regression. diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 4ac32be3..da0b6107 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -847,6 +847,40 @@ def test_absent_turning_points_only_warn(self) -> None: assert any("turning points" in w for w in warnings) +class TestUnscoredDebriefRemedyIsStillTrue: + """The unscored-debrief note explains *why* nothing asks for observations. + + That explanation is the whole value of the note -- it is what keeps triage + off the two wrong tracks (a parse failure, or a different scenario) -- and it + is a claim about product source this harness does not import. If the product + grows a rubric route into the turn prompt, the note starts handing out advice + for a weakness that no longer exists, and nothing else in this file notices. + """ + + _PROMPT_SRC = REPO_ROOT / "packages" / "prompt-composer" / "src" / "convsim_prompt" + + def test_no_prompt_layer_names_a_rubric_dimension(self) -> None: + layers = (self._PROMPT_SRC / "layers.py").read_text(encoding="utf-8") + assert "rubric" not in layers.lower(), ( + "a prompt layer now mentions a rubric, so UNSCORED_DEBRIEF_NOTE's " + "'no prompt layer names the rubric dimensions' may no longer hold" + ) + + def test_a_scenario_cannot_carry_a_rubric_into_the_turn_prompt(self) -> None: + # Why the note must NOT say "or play a scenario that defines one": the + # official job-interview-basic pack's behavioral_interview does define a + # rubric, and composing a turn prompt from it would change nothing, + # because the dataclass the composer is handed has nowhere to put it. + types_src = (self._PROMPT_SRC / "types.py").read_text(encoding="utf-8") + start = types_src.index("class ScenarioData:") + body = types_src[start:types_src.index("\n@", start)] + assert "rubric" not in body.lower(), ( + "ScenarioData now carries a rubric, so a rubric-defining scenario may " + "reach the turn prompt after all -- revisit UNSCORED_DEBRIEF_NOTE and " + "the 'Unscored debrief' section of docs/real-model-smoke.md" + ) + + # --------------------------------------------------------------------------- # Budget evaluation # --------------------------------------------------------------------------- From 146c0c7927ba29de9652d95a311d04274f46dfa5 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 08:57:54 -0400 Subject: [PATCH 47/58] Replace the runtime budget's invented measurements with the real ones MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The budget breakdown said its non-inference rows were measured. They were not: this job's own logs for the ten nightlies still in GitHub's retention disagree with every one of them by five to ten times. Measured, from those logs — the cold-cache rows from the one of the ten that missed the cache: set up job + checkout + setup-python ~10 s (doc said ~1 min) pip install x3 ~16 s (doc said ~2 min) model cache restore (2.3 GiB) ~25 s (no row at all) download + SHA-256 verify (2.3 GiB) ~9 s (doc said ~2 min + ~1 min) cache save (2.3 GiB) ~10 s (doc said ~1 min) model load + core start + session ~8 s (doc said ~0.5 min) So the claim that the steps before the smoke cost "up to ~9 min that the harness's own clock never sees" — the whole stated reason the harness budget is 20 min inside a 30 min job — is really ~1 min, and the ~15/~19 min totals carried about 5 min of phantom overhead on top of the two legitimately projected inference rows. The 20 min and 30 min ceilings are unchanged and still right; the reasoning printed next to them was false, and it is the reasoning the next person to tune them would start from. Two of the real numbers are counter-intuitive enough to state outright: a cache MISS (~9 s to fetch and hash the GGUF from Hugging Face) is cheaper than a cache HIT (~25 s to restore the same bytes from the Actions cache), so the cache saves the network and not the clock; and everything before the conversation together is ~1 min of a 30 min job, so the budget is inference and nothing else. The table now also carries the cache-restore row it was missing, and marks the one row that genuinely has no measurement — the harness's new re-verify before load — as bounded by the cold download row rather than measured. The inference rows stay projected, but from ten samples instead of one: 66/93/98/103/104/111/112/114/115/115 s for a single behavioral-interview turn. CI_HARDWARE_FACTOR's calibration prose quoted the top of that range (~116 s) as if it were typical; it now quotes the range and says the factor is set from the slow end deliberately. The 66 s run against the 115 s one is also the night-to-night spread a two-sample calibration could not have seen, which is the better argument for the 20x factor than any single measurement. Also attributes the replayed-opening evidence to its actual cause. All ten of those nightlies logged an NPC reply reciting the authored opening verbatim, which is what `replayed_opening` exists to catch — but the player turn they submitted was "Reply with exactly one sentence: Hello, I am ready.", an instruction to the model rather than anything an interview candidate says, and this harness replaces it with the three in-character SCRIPTED_PLAYER_TURNS the real starter model answered properly in local verification. Ten-for-ten with no cause attached reads as "the starter model always recites", which would have the first recital warning scrolled past as the documented status quo instead of read as new information. Co-Authored-By: Claude Opus 5 --- .github/workflows/model-smoke-nightly.yml | 60 ++++++---- .../src/content/docs/dev/real-model-smoke.md | 108 ++++++++++++------ docs/real-model-smoke.md | 108 ++++++++++++------ scripts/nightly-model-smoke.py | 31 +++-- tests/scripts/test_nightly_model_smoke.py | 24 ++-- 5 files changed, 223 insertions(+), 108 deletions(-) diff --git a/.github/workflows/model-smoke-nightly.yml b/.github/workflows/model-smoke-nightly.yml index ff65188d..c267beeb 100644 --- a/.github/workflows/model-smoke-nightly.yml +++ b/.github/workflows/model-smoke-nightly.yml @@ -35,27 +35,41 @@ # timeout-minutes: 30 is the hard ceiling; the script's own # --wall-clock-budget-s 1200 (20 min) trips first so the failure is attributed # to a phase instead of surfacing as an unattributable job cancellation. The -# 10 min gap is not slack: every step before the smoke (checkout, pip install, -# cache restore and, on a cache miss, a 2.5 GB download plus the cache save) -# runs against the same job clock and is invisible to the script's own -# deadline — up to ~9 min of it on a cold night. +# 10 min gap is deliberate margin, not an accounting of known work: the steps +# before the smoke (checkout, pip install, cache restore and, on a cache miss, +# the download plus the cache save) run against the same job clock and are +# invisible to the script's own deadline, but they measure ~1 min in total. +# The margin is sized for the network-bound part of that, which a slow +# Hugging Face or Actions-cache night can stretch by minutes. # -# checkout + setup-python ~1 min -# pip install (cached wheels) ~2 min -# model download (cache miss only) ~2 min — ~0 on a cache hit -# SHA-256 verification (2.5 GB, ×2 cold) ~1 min — ~0.5 min on a cache hit -# cache save (cache miss only) ~1 min — ~0 on a cache hit -# llama-server model load ~0.5 min -# 3 scripted turns + authored opening ~7 min — projected, see below +# set up job + checkout + setup-python ~10 s +# pip install ×3 (cached wheels) ~16 s cold, ~13 s warm +# model cache restore (2.3 GiB) ~25 s on a hit, <1 s on a miss +# download + SHA-256 verify (cache miss) ~9 s +# cache save (cache miss only) ~10 s +# SHA-256 re-verify before load <10 s — bounded, see below +# model load + core start + session ~8 s +# 3 scripted turns ~6 min — projected, see below # debrief generation ~4 min — projected, see below # ------------------------------------------------ -# total ~15 min warm cache, ~19 min cold +# total ~11 min, warm cache or cold # -# The download, verification, cache and model-load figures are measured; the -# inference figures are projected from the one latency this job has actually -# measured (~116 s per behavioral-interview turn, see below) and should be -# replaced with real per-phase durations from the first green nightly's report -# artifact. +# The non-inference figures are measured, from this job's own logs for the ten +# nightlies to 2026-10-03. Two of them are counter-intuitive and worth knowing +# before tuning anything: a cache MISS (~9 s to fetch and hash the GGUF) is +# cheaper than a cache HIT (~25 s to restore the same bytes), so the cache +# saves the network and not the clock; and everything before the conversation +# is ~1 min of a 30 min job, so the budget is inference and nothing else. +# The re-verify row is the one with no direct measurement — the previous +# single-turn harness did not re-verify — and is bounded by the cold row above, +# where downloading 2.3 GiB *and* hashing it together took 9 s. +# +# The inference figures are projected from the one latency this job has +# actually measured — a single behavioral-interview turn, ten nightlies: +# 66/93/98/103/104/111/112/114/115/115 s — rounded up to ~120 s per turn +# because turns 2 and 3 render a longer transcript than turn 1, with the +# debrief allowed twice a turn. Replace both with real per-phase durations +# from the first green nightly's report artifact. # # If the smoke step creeps past ~15 min — or the job past ~25 min — reduce # SCRIPTED_PLAYER_TURNS rather than raising either ceiling: a nightly that @@ -68,9 +82,10 @@ # CI budget = documented budget × CI_HARDWARE_FACTOR (currently 20) # # The factor is calibrated empirically: a full behavioral-interview turn on -# this runner measures ~116 s (two consecutive runs at 117.0 s and 115.3 s — -# large grammar-constrained prompt, 4B-Q4 model, CPU-only), i.e. ~11.6× the -# 10 s documented budget. 20× gives a 10 s × 20 = 200 s CI budget and, with +# this runner has measured 66–115 s across the ten nightlies to 2026-10-03 +# (median ~108 s — large grammar-constrained prompt, 4B-Q4 model, CPU-only), +# i.e. ~11.6× the 10 s documented budget at the slow end, which is the end the +# factor is set from. 20× gives a 10 s × 20 = 200 s CI budget and, with # the script's 1.20 regression tolerance, a 240 s ceiling — roughly 2× headroom # over the observed latency so normal runner-to-runner variance does not flap # the nightly, while a genuine >2× regression still fails. The headline @@ -125,8 +140,9 @@ jobs: # extra index — and --extra-index-url does not prefer them: pip resolves # the newest version across both indexes, so a release the wheel index has # not caught up with yet is installed by compiling llama.cpp on the - # runner. That build is the ~2 min row in the runtime budget turning into - # 15+, which the 30 min job timeout now has no room for: the job would die + # runner. That build turns the ~16 s install row in the runtime budget + # into 15+ min, which the 30 min job timeout has no room for next to ~11 + # min of inference: the job would die # with the unattributable cancellation this whole design exists to avoid, # on a night when nothing about the product changed. Refusing the sdist # makes pip fall back to the newest version that does have a wheel, and if diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index dec767be..37e7a0e0 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -57,15 +57,27 @@ The NPC *opening* line is authored scenario text, not a generation — so the scripted player turns, not the opening, are what prove the model is working. A turn that is real model output is not automatically a *reply*, and the -per-turn parse flags cannot tell the difference. Every nightly run of the -previous single-turn harness logged an NPC answer whose leading characters were -byte-identical to the scenario's `opening_npc_says` — the model reciting the -opening question back instead of answering it. `used_fallback` stays false for -those turns (the utterance is neither empty nor the canned safe one), so the run -went green on a conversation that never happened. The harness therefore records -`replayed_opening` per turn and applies the same policy it applies to -fallbacks: one such turn warns, *every* generated turn doing it fails, because -then the model answered nothing and the run proves nothing. +per-turn parse flags cannot tell the difference. All ten nightly runs of the +previous single-turn harness still in GitHub's log retention logged an NPC +answer whose leading characters were byte-identical to the scenario's +`opening_npc_says` — the model reciting the opening question back instead of +answering it. `used_fallback` stays false for those turns (the utterance is +neither empty nor the canned safe one), so the run went green on a conversation +that never happened. The harness therefore records `replayed_opening` per turn +and applies the same policy it applies to fallbacks: one such turn warns, +*every* generated turn doing it fails, because then the model answered nothing +and the run proves nothing. + +Read that ten-for-ten with its cause attached, because it is not evidence that +the starter model always recites. The single player turn those runs submitted +was `"Reply with exactly one sentence: Hello, I am ready."` — an instruction +addressed to the model rather than something an interview candidate says — and +this job replaces it with the three in-character turns in +`SCRIPTED_PLAYER_TURNS`, which the real starter model answered properly in local +verification. So the recital is a known failure mode of the *old* prompt, and +the baseline rate under the new script is simply not yet known: expect zero, and +treat the first recital warning as new information rather than as the documented +status quo. The two policies also compose, because a conversation can be *split* between them — one turn recites the opening, the others fall back — and trip neither @@ -239,32 +251,58 @@ Target: **under 30 minutes on a standard GitHub-hosted runner.** The job sets that ran long — a GitHub-side timeout would only say "the operation was canceled". -The 10-minute gap between the two is not slack. `timeout-minutes` covers the -whole job, and the harness only starts after checkout, three `pip install` -steps, the cache restore and — on a cache miss — a 2.5 GB download *and* the -cache save that follows it: up to ~9 min that the harness's own clock never -sees. A 25 min harness budget would lose the race to the job timeout on exactly -the cold-cache nights where an attributed verdict matters most. +The 10-minute gap between the two is deliberate margin, not an accounting of +known work. `timeout-minutes` covers the whole job, and the harness only starts +after checkout, three `pip install` steps and the model cache restore — which +together measure **~1 min** on the nightlies run so far, not the several +minutes a 2.5 GB transfer sounds like. The gap is sized for the part of that +work that is network-bound rather than for its measured cost: a slow +Hugging Face or Actions-cache night can cost minutes that the harness's own +clock never sees, and the whole design exists so that a slow night still +produces an attributed verdict rather than a job cancellation. On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | |---|---|---| -| Checkout + `setup-python` | ~1 min | ~1 min | -| `pip install` (prompt-composer, convsim-core, llama-cpp-python wheel) | ~2 min | ~2 min | -| Model download (2.5 GB from Hugging Face) | ~2 min | — | -| SHA-256 verification | ~1 min (×2) | ~0.5 min | -| Cache save (2.5 GB) | ~1 min | — | -| `llama-server` model load | ~0.5 min | ~0.5 min | -| Authored opening + 3 scripted turns † | ~7 min | ~7 min | +| Set up job + checkout + `setup-python` | ~10 s | ~10 s | +| `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~13 s | +| Model cache restore (2.3 GiB) | — (miss: <1 s) | ~25 s | +| Model download from Hugging Face + SHA-256 verify (2.3 GiB) | ~9 s | — | +| Cache save (2.3 GiB, post-step) | ~10 s | — | +| SHA-256 re-verify before the weights are loaded ‡ | <10 s | <10 s | +| `llama-server` model load + `convsim-core` start + session create | ~8 s | ~8 s | +| 3 scripted turns † | ~6 min | ~6 min | | Debrief generation † | ~4 min | ~4 min | -| **Total** | **~19 min** | **~15 min** | - -† The download, verification, cache and model-load rows are measured. The two -inference rows are *projected* from the only latency this job has measured so -far — ~116 s for a single behavioral-interview turn (see below) — because the -multi-turn conversation and the debrief have never run on a runner. Replace them -with the real `phase_durations_s` from the first green nightly's report artifact. +| **Total** | **~11 min** | **~11 min** | + +The non-inference rows are measured, from the job's own logs for the ten +nightlies to 2026-10-03 — the cold-cache rows from the one of those that missed +the cache. Two things in them are worth knowing before tuning anything: + +- **A cache miss is cheaper than a cache hit.** Downloading the GGUF from + Hugging Face and hashing it takes ~9 s; restoring the same bytes from the + Actions cache takes ~25 s. Caching the model saves the *network*, not the + clock, and abandoning a poisoned cache entry costs nothing. +- **Everything before the conversation is noise.** All of it together is ~1 min + of a 30 min job. The budget is inference, and only inference. + +‡ The harness re-verifies the on-disk file at the start of every run, which the +previous single-turn harness did not, so this row has no direct measurement. +The bound comes from the cold-cache row above: downloading 2.3 GiB *and* +hashing it took 9 s in total, so the hash alone is a few seconds. + +† The two inference rows are *projected*, because the multi-turn conversation +and the debrief have never run on a runner. The projection comes from the only +latency this job has measured — one behavioral-interview turn, ten nightlies: +66 / 93 / 98 / 103 / 104 / 111 / 112 / 114 / 115 / 115 s — rounded up to ~120 s +per turn, since turns 2 and 3 render a longer transcript than turn 1, and a +debrief allowed twice a turn (`DEBRIEF_SLOWDOWN_FACTOR`). Replace both rows with +the real `phase_durations_s` from the first green nightly's report artifact. + +At ~11 min projected against a 20 min harness budget there is roughly 2× +headroom, and the harness's clock covers only the rows from the re-verify +downwards. The model is cached between runs under the key `model-gguf-v1-`, so the download only recurs when the registry @@ -287,11 +325,13 @@ CI ceiling = documented budget × CI_HARDWARE_FACTOR × REGRESSION_TOLERANCE ``` `CI_HARDWARE_FACTOR` (20) is calibrated empirically: a full behavioral-interview -turn on this runner measures ~116 s, i.e. ~11.6× the 10 s documented budget. The -factor leaves roughly 2× headroom so runner-to-runner variance does not flap the -nightly, while a genuine >2× regression still fails. The headline -`full_response_ms` is the **median** of the scripted turns, so one unlucky turn -cannot flap the job either. +turn on this runner has measured 66–115 s across the ten nightlies to +2026-10-03 (median ~108 s), i.e. ~11.6× the 10 s documented budget at the slow +end. The factor is set from that slow end, not the median, and still leaves +roughly 2× headroom — note the 66 s run against the 115 s one, which is the +night-to-night spread a single-sample calibration would have missed. A genuine +>2× regression still fails. The headline `full_response_ms` is the **median** of +the scripted turns, so one unlucky turn cannot flap the job either. This factor models CI slowness only — the product's 10 s target-hardware budget is unchanged. Re-measure and re-tune it if the runner class or the starter model diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index eca2e8d9..3df2be3e 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -53,15 +53,27 @@ The NPC *opening* line is authored scenario text, not a generation — so the scripted player turns, not the opening, are what prove the model is working. A turn that is real model output is not automatically a *reply*, and the -per-turn parse flags cannot tell the difference. Every nightly run of the -previous single-turn harness logged an NPC answer whose leading characters were -byte-identical to the scenario's `opening_npc_says` — the model reciting the -opening question back instead of answering it. `used_fallback` stays false for -those turns (the utterance is neither empty nor the canned safe one), so the run -went green on a conversation that never happened. The harness therefore records -`replayed_opening` per turn and applies the same policy it applies to -fallbacks: one such turn warns, *every* generated turn doing it fails, because -then the model answered nothing and the run proves nothing. +per-turn parse flags cannot tell the difference. All ten nightly runs of the +previous single-turn harness still in GitHub's log retention logged an NPC +answer whose leading characters were byte-identical to the scenario's +`opening_npc_says` — the model reciting the opening question back instead of +answering it. `used_fallback` stays false for those turns (the utterance is +neither empty nor the canned safe one), so the run went green on a conversation +that never happened. The harness therefore records `replayed_opening` per turn +and applies the same policy it applies to fallbacks: one such turn warns, +*every* generated turn doing it fails, because then the model answered nothing +and the run proves nothing. + +Read that ten-for-ten with its cause attached, because it is not evidence that +the starter model always recites. The single player turn those runs submitted +was `"Reply with exactly one sentence: Hello, I am ready."` — an instruction +addressed to the model rather than something an interview candidate says — and +this job replaces it with the three in-character turns in +`SCRIPTED_PLAYER_TURNS`, which the real starter model answered properly in local +verification. So the recital is a known failure mode of the *old* prompt, and +the baseline rate under the new script is simply not yet known: expect zero, and +treat the first recital warning as new information rather than as the documented +status quo. The two policies also compose, because a conversation can be *split* between them — one turn recites the opening, the others fall back — and trip neither @@ -235,32 +247,58 @@ Target: **under 30 minutes on a standard GitHub-hosted runner.** The job sets that ran long — a GitHub-side timeout would only say "the operation was canceled". -The 10-minute gap between the two is not slack. `timeout-minutes` covers the -whole job, and the harness only starts after checkout, three `pip install` -steps, the cache restore and — on a cache miss — a 2.5 GB download *and* the -cache save that follows it: up to ~9 min that the harness's own clock never -sees. A 25 min harness budget would lose the race to the job timeout on exactly -the cold-cache nights where an attributed verdict matters most. +The 10-minute gap between the two is deliberate margin, not an accounting of +known work. `timeout-minutes` covers the whole job, and the harness only starts +after checkout, three `pip install` steps and the model cache restore — which +together measure **~1 min** on the nightlies run so far, not the several +minutes a 2.5 GB transfer sounds like. The gap is sized for the part of that +work that is network-bound rather than for its measured cost: a slow +Hugging Face or Actions-cache night can cost minutes that the harness's own +clock never sees, and the whole design exists so that a slow night still +produces an attributed verdict rather than a job cancellation. On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | |---|---|---| -| Checkout + `setup-python` | ~1 min | ~1 min | -| `pip install` (prompt-composer, convsim-core, llama-cpp-python wheel) | ~2 min | ~2 min | -| Model download (2.5 GB from Hugging Face) | ~2 min | — | -| SHA-256 verification | ~1 min (×2) | ~0.5 min | -| Cache save (2.5 GB) | ~1 min | — | -| `llama-server` model load | ~0.5 min | ~0.5 min | -| Authored opening + 3 scripted turns † | ~7 min | ~7 min | +| Set up job + checkout + `setup-python` | ~10 s | ~10 s | +| `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~13 s | +| Model cache restore (2.3 GiB) | — (miss: <1 s) | ~25 s | +| Model download from Hugging Face + SHA-256 verify (2.3 GiB) | ~9 s | — | +| Cache save (2.3 GiB, post-step) | ~10 s | — | +| SHA-256 re-verify before the weights are loaded ‡ | <10 s | <10 s | +| `llama-server` model load + `convsim-core` start + session create | ~8 s | ~8 s | +| 3 scripted turns † | ~6 min | ~6 min | | Debrief generation † | ~4 min | ~4 min | -| **Total** | **~19 min** | **~15 min** | - -† The download, verification, cache and model-load rows are measured. The two -inference rows are *projected* from the only latency this job has measured so -far — ~116 s for a single behavioral-interview turn (see below) — because the -multi-turn conversation and the debrief have never run on a runner. Replace them -with the real `phase_durations_s` from the first green nightly's report artifact. +| **Total** | **~11 min** | **~11 min** | + +The non-inference rows are measured, from the job's own logs for the ten +nightlies to 2026-10-03 — the cold-cache rows from the one of those that missed +the cache. Two things in them are worth knowing before tuning anything: + +- **A cache miss is cheaper than a cache hit.** Downloading the GGUF from + Hugging Face and hashing it takes ~9 s; restoring the same bytes from the + Actions cache takes ~25 s. Caching the model saves the *network*, not the + clock, and abandoning a poisoned cache entry costs nothing. +- **Everything before the conversation is noise.** All of it together is ~1 min + of a 30 min job. The budget is inference, and only inference. + +‡ The harness re-verifies the on-disk file at the start of every run, which the +previous single-turn harness did not, so this row has no direct measurement. +The bound comes from the cold-cache row above: downloading 2.3 GiB *and* +hashing it took 9 s in total, so the hash alone is a few seconds. + +† The two inference rows are *projected*, because the multi-turn conversation +and the debrief have never run on a runner. The projection comes from the only +latency this job has measured — one behavioral-interview turn, ten nightlies: +66 / 93 / 98 / 103 / 104 / 111 / 112 / 114 / 115 / 115 s — rounded up to ~120 s +per turn, since turns 2 and 3 render a longer transcript than turn 1, and a +debrief allowed twice a turn (`DEBRIEF_SLOWDOWN_FACTOR`). Replace both rows with +the real `phase_durations_s` from the first green nightly's report artifact. + +At ~11 min projected against a 20 min harness budget there is roughly 2× +headroom, and the harness's clock covers only the rows from the re-verify +downwards. The model is cached between runs under the key `model-gguf-v1-`, so the download only recurs when the registry @@ -283,11 +321,13 @@ CI ceiling = documented budget × CI_HARDWARE_FACTOR × REGRESSION_TOLERANCE ``` `CI_HARDWARE_FACTOR` (20) is calibrated empirically: a full behavioral-interview -turn on this runner measures ~116 s, i.e. ~11.6× the 10 s documented budget. The -factor leaves roughly 2× headroom so runner-to-runner variance does not flap the -nightly, while a genuine >2× regression still fails. The headline -`full_response_ms` is the **median** of the scripted turns, so one unlucky turn -cannot flap the job either. +turn on this runner has measured 66–115 s across the ten nightlies to +2026-10-03 (median ~108 s), i.e. ~11.6× the 10 s documented budget at the slow +end. The factor is set from that slow end, not the median, and still leaves +roughly 2× headroom — note the 66 s run against the 115 s one, which is the +night-to-night spread a single-sample calibration would have missed. A genuine +>2× regression still fails. The headline `full_response_ms` is the **median** of +the scripted turns, so one unlucky turn cannot flap the job either. This factor models CI slowness only — the product's 10 s target-hardware budget is unchanged. Re-measure and re-tune it if the runner class or the starter model diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 2708e71e..39b33b37 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -146,10 +146,13 @@ class exit meaning # GitHub killing the job with an unattributable "The operation was canceled", # so this has to clear the job's timeout-minutes *minus the steps that run # before the script does*: checkout, pip install, cache restore and (on a cache -# miss) the 2.5 GB download plus the cache save that follows it, together up to -# ~9 min of the 30 min job. 20 min therefore leaves the script's own deadline -# the first one to trip. -# See docs/real-model-smoke.md. +# miss) the download plus the cache save that follows it. Those measure ~1 min +# in total on the nightlies run so far — not the several minutes a 2.5 GB +# transfer sounds like — so 20 min inside a 30 min job leaves ~9 min of real +# margin for the network-bound part of them, and leaves this deadline the first +# one to trip. Against the ~11 min the run itself is projected to take, 20 min +# is also roughly 2× headroom. +# See docs/real-model-smoke.md for the measured breakdown. DEFAULT_WALL_CLOCK_BUDGET_S = 1200.0 # 20 min # Scripted player turns — deliberately the same script as the fake-runtime @@ -958,14 +961,24 @@ def _replays_opening(npc_text: str, opening_text: str) -> bool: """True when an NPC reply opens by reciting the authored opening verbatim. The scenario's opening line sits in the transcript the turn prompt renders, - and the starter model answers the first player turn by copying it back: - every nightly run of the previous single-turn harness logged an NPC reply - whose leading characters matched ``opening_npc_says`` exactly. That is not - a reply, but nothing downstream notices — the utterance is non-empty and it - is not ``SAFE_FALLBACK_UTTERANCE``, so ``used_fallback`` stays false and the + and the starter model can answer a player turn by copying it back: all ten + nightly runs of the previous single-turn harness still in GitHub's log + retention logged an NPC reply whose leading characters matched + ``opening_npc_says`` exactly. That is not a reply, but nothing downstream + notices — the utterance is non-empty and it is not + ``SAFE_FALLBACK_UTTERANCE``, so ``used_fallback`` stays false and the per-turn parse flags report a healthy turn. Proving the turns are real therefore needs this check as well as those flags. + Those ten runs are evidence that the failure mode exists, not that it is the + norm: the player turn they submitted was "Reply with exactly one sentence: + Hello, I am ready." — an instruction to the model, not something an + interview candidate says — and this harness replaces it with the + in-character ``SCRIPTED_PLAYER_TURNS``, which the real starter model + answered properly in local verification. So the expected rate here is zero + and the warning is worth reading when it fires, rather than a known + condition to scroll past. + A *leading prefix* match, not equality and not the whole opening: a recital that then carries on into fresh prose is the same regurgitation, and a recital that veers off partway through the opening is too — see diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index da0b6107..2a3bae6d 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -1017,10 +1017,15 @@ def test_the_enforced_budget_is_published_in_the_performance_doc( # --------------------------------------------------------------------------- -# Steps that burn the job's clock before the harness starts and so are invisible -# to its own deadline: checkout, three pip installs, the cache restore and, on a -# cache miss, the 2.5 GB download plus the cache save that follows it. ~9 min -# cold; see the breakdown in docs/real-model-smoke.md. +# Allowance for the steps that burn the job's clock before the harness starts and +# so are invisible to its own deadline: checkout, three pip installs, the cache +# restore and, on a cache miss, the download plus the cache save that follows it. +# +# Deliberately far above the measurement. Those steps total ~1 min on the +# nightlies run so far (see the measured breakdown in docs/real-model-smoke.md), +# but they are the network-bound part of the job, so this asserts the two +# ceilings stay compatible with a night on which they run several times slower. +# Lowering it to the measured ~1 min would make the assertion vacuous. PRE_SMOKE_JOB_MINUTES = 9 _WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "model-smoke-nightly.yml" @@ -1166,11 +1171,12 @@ class TestWheelInstallInvariants: PyPI ships llama-cpp-python as an sdist only — the CPU wheels live on the extra index — and ``--extra-index-url`` does not *prefer* them: pip resolves the newest version across both indexes, so on any night the wheel index lags - PyPI, pip compiles llama.cpp on the runner. That is the ~2 min install row - of the runtime budget turning into 15+, which the 30 min job timeout has no - room for: the job dies with exactly the unattributable "operation was - canceled" that the harness's phase-attributed deadline exists to replace, on - a night when nothing about the product changed. + PyPI, pip compiles llama.cpp on the runner. That turns the ~16 s install + row of the runtime budget into 15+ min, which the 30 min job timeout has no + room for next to ~11 min of inference: the job dies with exactly the + unattributable "operation was canceled" that the harness's phase-attributed + deadline exists to replace, on a night when nothing about the product + changed. Like the checksum promises above, this lives in the YAML and nowhere else, and it reads as a redundant flag next to the extra index that is "obviously" From b3591d3b6d238409c444b84da4c985a1c8ceab0b Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 09:12:22 -0400 Subject: [PATCH 48/58] Stop the model download from burying the verdict printed beneath it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The progress line is written with a leading "\r" and no newline, which a terminal collapses in place for free. CI has no terminal: it captures stdout through a pipe, so every explicit flush becomes its own log line. The one nightly still in retention that missed the model cache logged 2381 of them for a 2.3 GiB GGUF — and a `download` or `checksum` verdict is printed directly underneath that, which is the output a triager has to scroll past to reach the thing this job exists to tell them. Throttle to one line per PROGRESS_LOG_STEP_PCT (10 %). Eleven lines still show where a stalled transfer stopped; 2381 hide the banner they precede. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 23 +++++++++++---- tests/scripts/test_nightly_model_smoke.py | 34 +++++++++++++++++++++++ 2 files changed, 52 insertions(+), 5 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 39b33b37..7f43f1b4 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -170,6 +170,16 @@ class exit meaning # Longest NPC excerpt copied into the log / report (scripted content only). EXCERPT_CHARS = 160 +# How much of the download has to complete before another progress line is +# printed. A terminal collapses the "\r" updates in place, so per-megabyte +# progress costs nothing there — but CI captures this script through a pipe with +# no terminal, where every flush becomes its own log line. The one nightly in +# retention that missed the model cache logged 2381 of them for a 2.3 GiB GGUF, +# which is the wall of output a reader has to scroll past to reach the +# `download` or `checksum` verdict printed directly underneath it. Eleven lines +# still show a stalled transfer; 2381 hide the verdict they precede. +PROGRESS_LOG_STEP_PCT = 10 + # A debrief summary shorter than this is not a usable coaching summary. MIN_SUMMARY_CHARS = 20 @@ -607,6 +617,7 @@ def _download_with_progress(url: str, dest: Path) -> None: total = int(resp.headers.get("Content-Length", 0)) or None downloaded = 0 chunk = 1 << 20 # 1 MB + last_logged_pct = -PROGRESS_LOG_STEP_PCT while True: buf = resp.read(chunk) if not buf: @@ -615,11 +626,13 @@ def _download_with_progress(url: str, dest: Path) -> None: downloaded += len(buf) if total: pct = int(downloaded / total * 100) - print( - f"\r {pct}% ({downloaded // 1_048_576} / {total // 1_048_576} MB)", - end="", - flush=True, - ) + if pct - last_logged_pct >= PROGRESS_LOG_STEP_PCT: + last_logged_pct = pct + print( + f"\r {pct}% ({downloaded // 1_048_576} / {total // 1_048_576} MB)", + end="", + flush=True, + ) print() except Exception as exc: # noqa: BLE001 — see below # Network failure, HTTP error, or a full disk — all "could not fetch the diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 2a3bae6d..0616bbb3 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -249,6 +249,40 @@ def test_an_unrequestable_url_is_not_reported_as_a_harness_bug( assert "--model-url" in exc_info.value.remedy assert "registry" in exc_info.value.remedy + def test_progress_is_logged_sparsely_not_once_per_megabyte( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture + ) -> None: + # The "\r" updates collapse in a terminal but not in a CI log, where + # each flush is its own line: the one nightly that missed the model + # cache logged 2381 of them for a 2.3 GiB GGUF, directly above the + # verdict a triager is looking for. + megabytes = 200 + chunks = [b"\0" * (1 << 20)] * megabytes + + class _Resp: + headers = {"Content-Length": str(megabytes << 20)} + + def read(self, _n: int) -> bytes: + return chunks.pop(0) if chunks else b"" + + def __enter__(self): + return self + + def __exit__(self, *a: object) -> bool: + return False + + monkeypatch.setattr(smoke.urllib.request, "urlopen", lambda *a, **k: _Resp()) + smoke._download_with_progress("https://example.invalid/m.gguf", tmp_path / "m.gguf") + + progress_lines = [ + line for line in capsys.readouterr().out.split("\r") if line.strip().endswith("MB)") + ] + # One per PROGRESS_LOG_STEP_PCT, plus the 0 % line: nowhere near one per MB. + assert len(progress_lines) <= 100 // smoke.PROGRESS_LOG_STEP_PCT + 1 + # Still enough to show where a stalled transfer stopped. + assert len(progress_lines) >= 2 + assert "100%" in progress_lines[-1] + # --------------------------------------------------------------------------- # Registry resolution From 57d26ca4acd920552fd532979e26032b33719cb9 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 09:12:31 -0400 Subject: [PATCH 49/58] Stop the budget table from billing the cache save to a step that no longer exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runtime-budget breakdown still labels the cache save a "post-step", which it was under `actions/cache`. This PR split restore and save precisely so the save could be gated on `success()`, and that made it an ordinary step that runs between the download and the smoke — two paragraphs below the table the doc explains the split and the gating. A reader checking the ~10 s row against a job timeline would go looking for a post step that is not there, and the ordering the gating depends on is the opposite of what "post-step" implies. The prose above the table has the same gap: it enumerates what runs before the harness as "checkout, three pip install steps and the model cache restore", which is the warm-cache job. A cold run also pays the download and the save before the harness starts, and the ~1 min total it claims is only true because both of those are counted. Co-Authored-By: Claude Opus 5 --- docs-site/src/content/docs/dev/real-model-smoke.md | 7 ++++--- docs/real-model-smoke.md | 7 ++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 37e7a0e0..11dbeaf4 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -253,8 +253,9 @@ canceled". The 10-minute gap between the two is deliberate margin, not an accounting of known work. `timeout-minutes` covers the whole job, and the harness only starts -after checkout, three `pip install` steps and the model cache restore — which -together measure **~1 min** on the nightlies run so far, not the several +after checkout, three `pip install` steps, the model cache restore and — on a +cache miss — the download and the cache save that follows it, which together +measure **~1 min** on the nightlies run so far, not the several minutes a 2.5 GB transfer sounds like. The gap is sized for the part of that work that is network-bound rather than for its measured cost: a slow Hugging Face or Actions-cache night can cost minutes that the harness's own @@ -269,7 +270,7 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~13 s | | Model cache restore (2.3 GiB) | — (miss: <1 s) | ~25 s | | Model download from Hugging Face + SHA-256 verify (2.3 GiB) | ~9 s | — | -| Cache save (2.3 GiB, post-step) | ~10 s | — | +| Cache save (2.3 GiB, its own step before the smoke) | ~10 s | — | | SHA-256 re-verify before the weights are loaded ‡ | <10 s | <10 s | | `llama-server` model load + `convsim-core` start + session create | ~8 s | ~8 s | | 3 scripted turns † | ~6 min | ~6 min | diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 3df2be3e..54bfd057 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -249,8 +249,9 @@ canceled". The 10-minute gap between the two is deliberate margin, not an accounting of known work. `timeout-minutes` covers the whole job, and the harness only starts -after checkout, three `pip install` steps and the model cache restore — which -together measure **~1 min** on the nightlies run so far, not the several +after checkout, three `pip install` steps, the model cache restore and — on a +cache miss — the download and the cache save that follows it, which together +measure **~1 min** on the nightlies run so far, not the several minutes a 2.5 GB transfer sounds like. The gap is sized for the part of that work that is network-bound rather than for its measured cost: a slow Hugging Face or Actions-cache night can cost minutes that the harness's own @@ -265,7 +266,7 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~13 s | | Model cache restore (2.3 GiB) | — (miss: <1 s) | ~25 s | | Model download from Hugging Face + SHA-256 verify (2.3 GiB) | ~9 s | — | -| Cache save (2.3 GiB, post-step) | ~10 s | — | +| Cache save (2.3 GiB, its own step before the smoke) | ~10 s | — | | SHA-256 re-verify before the weights are loaded ‡ | <10 s | <10 s | | `llama-server` model load + `convsim-core` start + session create | ~8 s | ~8 s | | 3 scripted turns † | ~6 min | ~6 min | From c4a15c2b67413e82c66489b7ee1954ee08c6c374 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 09:31:11 -0400 Subject: [PATCH 50/58] Carry the Milestone-1 tick through to the public roadmap and the deferral list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ROADMAP.md marks #457 done, but the two documents that mirror it were left saying the opposite: * docs-site/.../project/roadmap.md is a line-for-line mirror of ROADMAP.md and is the copy the public docs site renders. It still showed the item unticked, so the published roadmap contradicted the repository's own. The #455 commit (6248a4a9) updated both copies in one change; this one had updated only the repository copy. * post-alpha-issues.md item 6 is the deferral record for this very issue, and still presented it as deferred to Milestone 2 with the rationale "the fake runtime provides full structural coverage; real-model CI is a quality-of-life improvement" and no outcome. A reader of that page would conclude the nightly does not exist. It now carries a Status line in the same form items 2 and 3 use for shipped work, saying what the job does and pointing at the new doc — with the Milestone and Why-deferred lines left alone, since those are the historical record of the deferral rather than a claim about today. Docs only; no behaviour change. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/post-alpha-issues.md | 14 +++++++++++++- docs-site/src/content/docs/project/roadmap.md | 2 +- docs/post-alpha-issues.md | 11 +++++++++++ 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/docs-site/src/content/docs/dev/post-alpha-issues.md b/docs-site/src/content/docs/dev/post-alpha-issues.md index bce37ff4..3121404f 100644 --- a/docs-site/src/content/docs/dev/post-alpha-issues.md +++ b/docs-site/src/content/docs/dev/post-alpha-issues.md @@ -124,7 +124,19 @@ latency and output quality signals. cache-unfriendly in most CI environments. The fake runtime provides full structural coverage; real-model CI is a quality-of-life improvement. -**Milestone:** 2 (CI hardening) +**Milestone:** 2 (CI hardening) +**Status:** ✅ **Shipped,** pulled forward into the Milestone-1 polish set +rather than waiting for Milestone 2. +[`.github/workflows/model-smoke-nightly.yml`](https://github.com/outrightmental/ConversationSimulator/blob/main/.github/workflows/model-smoke-nightly.yml) +runs nightly: it resolves the registry's `starter` model from one lookup, +downloads it with SHA-256 verification (cached between runs and re-verified +before the weights load, so a drifted cache hit fails loudly), starts +`llama-server` and `convsim-core` on it, plays a scripted multi-turn +conversation and asserts a *scored* debrief — inside a documented budget under +30 min. Every failure is attributed to one of six classes with its own exit +code, so a red nightly is triageable from the job summary; the harness's own +decision logic is unit-tested per-PR. See +[Real-model smoke test](/dev/real-model-smoke/). --- diff --git a/docs-site/src/content/docs/project/roadmap.md b/docs-site/src/content/docs/project/roadmap.md index 162c546f..0049a6b4 100644 --- a/docs-site/src/content/docs/project/roadmap.md +++ b/docs-site/src/content/docs/project/roadmap.md @@ -77,7 +77,7 @@ criteria. - [x] Real UI screenshots and demo recording, captured from a local-model playthrough - [ ] Desktop app with bundled backend (Tauri sidecar for `convsim-core`) -- [ ] Automated real-model CI smoke test +- [x] Automated real-model CI smoke test ### Post-alpha — Milestone 2+ diff --git a/docs/post-alpha-issues.md b/docs/post-alpha-issues.md index 0d4ddc2c..c9f319e1 100644 --- a/docs/post-alpha-issues.md +++ b/docs/post-alpha-issues.md @@ -130,6 +130,17 @@ cache-unfriendly in most CI environments. The fake runtime provides full structural coverage; real-model CI is a quality-of-life improvement. **Milestone:** 2 (CI hardening) +**Status:** ✅ **Shipped** ([#457](https://github.com/outrightmental/ConversationSimulator/issues/457)), +pulled forward into the Milestone-1 polish set rather than waiting for +Milestone 2. `.github/workflows/model-smoke-nightly.yml` runs nightly: it +resolves the registry's `starter` model from one lookup, downloads it with +SHA-256 verification (cached between runs and re-verified before the weights +load, so a drifted cache hit fails loudly), starts `llama-server` and +`convsim-core` on it, plays a scripted multi-turn conversation and asserts a +*scored* debrief — inside a documented budget under 30 min. Every failure is +attributed to one of six classes with its own exit code, so a red nightly is +triageable from the job summary; the harness's own decision logic is unit-tested +per-PR. See [real-model-smoke.md](real-model-smoke.md). **Tracking:** [#457](https://github.com/outrightmental/ConversationSimulator/issues/457) --- From 04c615cc040b61855f34de06c490b55dad8185e1 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 09:31:55 -0400 Subject: [PATCH 51/58] Bound the model download so a slow night still yields an attributed verdict MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The download runs in its own step before the smoke, so it spends the job clock where the harness's Deadline cannot see it — and `urlopen`'s timeout is per socket read, not per transfer. A connection that went quiet repeatedly, or merely crawled, could therefore spend the whole ~9 min of margin between the harness's 20 min budget and the job's 30 min timeout and then hand the smoke a job clock with no room left in it. GitHub would cancel the job mid-conversation with no class, no phase and no remedy: the one unattributable outcome every other budget in this harness exists to replace, and the only route to it left open. The transfer now carries a 5 min budget (~33x the measured 9 s for 2.3 GiB, leaving ~4 min of the margin untouched) checked per chunk, with any single quiet stretch capped at 2 min. Exceeding either is exit 2 — `download`'s "nothing about the app changed, re-run the job" is the right advice for a stalled transfer — with its own remedy saying why the bound exists, and the partial file is still removed so the next run does not checksum-fail on it. Tests cover a crawling transfer (classified, own remedy, no partial file left), the read timeout sitting inside the overall budget, and the budget fitting inside the pre-smoke margin the workflow reserves. Co-Authored-By: Claude Opus 5 --- .github/workflows/model-smoke-nightly.yml | 9 ++- .../src/content/docs/dev/real-model-smoke.md | 14 +++- docs/real-model-smoke.md | 14 +++- scripts/nightly-model-smoke.py | 64 +++++++++++++++++- tests/scripts/test_nightly_model_smoke.py | 65 +++++++++++++++++++ 5 files changed, 156 insertions(+), 10 deletions(-) diff --git a/.github/workflows/model-smoke-nightly.yml b/.github/workflows/model-smoke-nightly.yml index c267beeb..51fb3799 100644 --- a/.github/workflows/model-smoke-nightly.yml +++ b/.github/workflows/model-smoke-nightly.yml @@ -21,7 +21,7 @@ # summary without reading logs: # # 1 budget — pipeline worked, latency regressed -# 2 download — model could not be fetched (network/HTTP/empty cache) +# 2 download — model could not be fetched (network/HTTP/stall/empty cache) # 3 checksum — SHA-256 drift against the registry # 4 runtime — llama-server or convsim-core crashed, hung, or 5xx'd # 5 pipeline — servers healthy, end-to-end assertion failed @@ -40,12 +40,15 @@ # the download plus the cache save) run against the same job clock and are # invisible to the script's own deadline, but they measure ~1 min in total. # The margin is sized for the network-bound part of that, which a slow -# Hugging Face or Actions-cache night can stretch by minutes. +# Hugging Face or Actions-cache night can stretch by minutes. The download is +# the one of those steps that could stretch without limit, so it carries its own +# 5 min budget (DOWNLOAD_BUDGET_S in the script) and fails as `download` rather +# than spending the margin the smoke still needs. # # set up job + checkout + setup-python ~10 s # pip install ×3 (cached wheels) ~16 s cold, ~13 s warm # model cache restore (2.3 GiB) ~25 s on a hit, <1 s on a miss -# download + SHA-256 verify (cache miss) ~9 s +# download + SHA-256 verify (cache miss) ~9 s — bounded at 5 min # cache save (cache miss only) ~10 s # SHA-256 re-verify before load <10 s — bounded, see below # model load + core start + session ~8 s diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 11dbeaf4..e4450e94 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -116,7 +116,7 @@ log, and a remedy printed next to it: | Exit | Class | Means | First thing to do | |---|---|---|---| | 1 | `budget` | Pipeline worked; latency regressed past the CI ceiling | Compare `measured_ms` in the report artifact against recent nightlies | -| 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | +| 2 | `download` | Model could not be fetched — network, HTTP, a transfer that stalled past its 5 min budget, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | | 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner. A port conflict started no child and so has no tail: the banner carries its own remedy instead, naming the port to free — see [Running it locally](#running-it-locally) | | 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect the per-turn `used_fallback` and `replayed_opening` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | @@ -269,7 +269,7 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Set up job + checkout + `setup-python` | ~10 s | ~10 s | | `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~13 s | | Model cache restore (2.3 GiB) | — (miss: <1 s) | ~25 s | -| Model download from Hugging Face + SHA-256 verify (2.3 GiB) | ~9 s | — | +| Model download from Hugging Face + SHA-256 verify (2.3 GiB) ◊ | ~9 s | — | | Cache save (2.3 GiB, its own step before the smoke) | ~10 s | — | | SHA-256 re-verify before the weights are loaded ‡ | <10 s | <10 s | | `llama-server` model load + `convsim-core` start + session create | ~8 s | ~8 s | @@ -288,6 +288,16 @@ the cache. Two things in them are worth knowing before tuning anything: - **Everything before the conversation is noise.** All of it together is ~1 min of a 30 min job. The budget is inference, and only inference. +◊ The download is the one pre-smoke step that could stretch without limit, and +it runs where the harness's own deadline cannot see it, so it carries its own +5 min budget (`DOWNLOAD_BUDGET_S`) with a 2 min cap on any single quiet stretch +(`DOWNLOAD_READ_TIMEOUT_S`). `urlopen`'s timeout is per socket read, not per +transfer, so a connection that goes quiet repeatedly — or merely crawls — would +otherwise spend the whole margin above and then hand the smoke a job clock with +no room in it, which is the one way left to reach an unattributable +cancellation. Exceeding either bound is exit 2 with its own remedy. 5 min is +~33× the measured 9 s, so a transfer has to be pathologically slow to hit it. + ‡ The harness re-verifies the on-disk file at the start of every run, which the previous single-turn harness did not, so this row has no direct measurement. The bound comes from the cold-cache row above: downloading 2.3 GiB *and* diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 54bfd057..978b83df 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -112,7 +112,7 @@ log, and a remedy printed next to it: | Exit | Class | Means | First thing to do | |---|---|---|---| | 1 | `budget` | Pipeline worked; latency regressed past the CI ceiling | Compare `measured_ms` in the report artifact against recent nightlies | -| 2 | `download` | Model could not be fetched — network, HTTP, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | +| 2 | `download` | Model could not be fetched — network, HTTP, a transfer that stalled past its 5 min budget, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | | 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner. A port conflict started no child and so has no tail: the banner carries its own remedy instead, naming the port to free — see [Running it locally](#running-it-locally) | | 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect the per-turn `used_fallback` and `replayed_opening` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | @@ -265,7 +265,7 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Set up job + checkout + `setup-python` | ~10 s | ~10 s | | `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~13 s | | Model cache restore (2.3 GiB) | — (miss: <1 s) | ~25 s | -| Model download from Hugging Face + SHA-256 verify (2.3 GiB) | ~9 s | — | +| Model download from Hugging Face + SHA-256 verify (2.3 GiB) ◊ | ~9 s | — | | Cache save (2.3 GiB, its own step before the smoke) | ~10 s | — | | SHA-256 re-verify before the weights are loaded ‡ | <10 s | <10 s | | `llama-server` model load + `convsim-core` start + session create | ~8 s | ~8 s | @@ -284,6 +284,16 @@ the cache. Two things in them are worth knowing before tuning anything: - **Everything before the conversation is noise.** All of it together is ~1 min of a 30 min job. The budget is inference, and only inference. +◊ The download is the one pre-smoke step that could stretch without limit, and +it runs where the harness's own deadline cannot see it, so it carries its own +5 min budget (`DOWNLOAD_BUDGET_S`) with a 2 min cap on any single quiet stretch +(`DOWNLOAD_READ_TIMEOUT_S`). `urlopen`'s timeout is per socket read, not per +transfer, so a connection that goes quiet repeatedly — or merely crawls — would +otherwise spend the whole margin above and then hand the smoke a job clock with +no room in it, which is the one way left to reach an unattributable +cancellation. Exceeding either bound is exit 2 with its own remedy. 5 min is +~33× the measured 9 s, so a transfer has to be pathologically slow to hit it. + ‡ The harness re-verifies the on-disk file at the start of every run, which the previous single-turn harness did not, so this row has no direct measurement. The bound comes from the cold-cache row above: downloading 2.3 GiB *and* diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 7f43f1b4..5de627eb 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -42,7 +42,8 @@ class exit meaning =========== ==== ======================================================= ``budget`` 1 latency regression: the pipeline worked, just too slowly -``download`` 2 model could not be fetched (network / HTTP / missing cache) +``download`` 2 model could not be fetched (network / HTTP / stalled + transfer / missing cache) ``checksum`` 3 SHA-256 drift: on-disk bytes ≠ registry ``sha256`` ``runtime`` 4 llama-server or convsim-core crashed, hung, or 5xx'd ``pipeline`` 5 servers healthy but an end-to-end assertion failed @@ -155,6 +156,27 @@ class exit meaning # See docs/real-model-smoke.md for the measured breakdown. DEFAULT_WALL_CLOCK_BUDGET_S = 1200.0 # 20 min +# Budget for the whole model transfer, in seconds, and for one quiet stretch of +# it. The download runs in its own `--download-only` step, *before* the smoke, +# so it burns the job clock where the Deadline above cannot see it — which makes +# an unbounded transfer the one remaining route to the unattributable "The +# operation was canceled" that every other budget here exists to replace. +# +# urllib's timeout is per socket operation, not per transfer, so the 600 s it +# used to be carried bounded nothing: a connection that went quiet repeatedly, +# or merely crawled, could spend the entire ~9 min of margin between the +# harness's 20 min and the job's 30 min and then hand the smoke a job clock with +# no room left in it. Both numbers are therefore sized against that margin +# rather than against the transfer: 300 s is ~33× the 9 s the measured +# nightlies take to fetch *and* hash 2.3 GiB, and leaves ~4 min of the margin +# untouched, while 120 s of silence is already far past anything a live +# Hugging Face connection does. +# +# Exceeding either is a `download` failure, whose remedy — the registry URL +# check, then re-run — is the right advice for a transfer that stalled. +DOWNLOAD_BUDGET_S = 300.0 +DOWNLOAD_READ_TIMEOUT_S = 120.0 + # Scripted player turns — deliberately the same script as the fake-runtime # playthrough in tests/e2e/test_scripted_playthrough.py (run at release time by # scripts/release-smoke.sh), so the two smokes exercise the same conversation @@ -579,7 +601,9 @@ def _emit_github_output(values: Dict[str, Any]) -> None: # ── Download and checksum ───────────────────────────────────────────────────── -def _download_with_progress(url: str, dest: Path) -> None: +def _download_with_progress( + url: str, dest: Path, *, budget_s: float = DOWNLOAD_BUDGET_S +) -> None: dest.parent.mkdir(parents=True, exist_ok=True) print(f" Downloading {url[:80]}…") try: @@ -612,8 +636,11 @@ def _download_with_progress(url: str, dest: Path) -> None: "scripts/validate-registry.py --url-check." ), ) from exc + started = time.monotonic() try: - with urllib.request.urlopen(req, timeout=600) as resp, open(dest, "wb") as f: + with urllib.request.urlopen( + req, timeout=DOWNLOAD_READ_TIMEOUT_S + ) as resp, open(dest, "wb") as f: total = int(resp.headers.get("Content-Length", 0)) or None downloaded = 0 chunk = 1 << 20 # 1 MB @@ -624,6 +651,30 @@ def _download_with_progress(url: str, dest: Path) -> None: break f.write(buf) downloaded += len(buf) + elapsed = time.monotonic() - started + if elapsed > budget_s: + # Checked per chunk rather than left to urllib's per-read + # timeout, which a transfer that merely crawls never trips. + # See DOWNLOAD_BUDGET_S: past this the job clock, not the + # transfer, is what runs out next. + rate = downloaded / elapsed / 1_048_576 if elapsed else 0.0 + raise SmokeFailure( + FailureClass.DOWNLOAD, + f"Download exceeded its {budget_s:.0f} s budget after " + f"{downloaded // 1_048_576} MB" + + (f" of {total // 1_048_576} MB" if total else "") + + f" ({rate:.1f} MB/s)", + remedy=( + "The transfer stalled or crawled rather than failing. " + "It is bounded because it runs before the smoke and on " + "the same job clock, which the harness's own deadline " + "cannot see: letting it run on would get the job " + "cancelled with no attributed verdict at all. Nothing " + "about the app changed — check that Hugging Face is " + "reachable (scripts/validate-registry.py --url-check) " + "and re-run the job." + ), + ) if total: pct = int(downloaded / total * 100) if pct - last_logged_pct >= PROGRESS_LOG_STEP_PCT: @@ -634,6 +685,13 @@ def _download_with_progress(url: str, dest: Path) -> None: flush=True, ) print() + except SmokeFailure: + # The exhausted-budget failure above, which already carries its own + # message and remedy. The partial file still has to go, but re-wrapping + # it in the generic verdict below would replace both with "Download + # failed for : SmokeFailure(...)". + dest.unlink(missing_ok=True) + raise except Exception as exc: # noqa: BLE001 — see below # Network failure, HTTP error, or a full disk — all "could not fetch the # model", none of which say anything about the app under test. Remove the diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 0616bbb3..d1287813 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -249,6 +249,53 @@ def test_an_unrequestable_url_is_not_reported_as_a_harness_bug( assert "--model-url" in exc_info.value.remedy assert "registry" in exc_info.value.remedy + def test_a_crawling_transfer_is_bounded_and_classified( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + # A transfer that keeps delivering bytes, just far too slowly, never + # trips urllib's per-read timeout. The download runs in its own step + # before the smoke, on a job clock the harness's Deadline cannot see, so + # without its own budget this is the one remaining way to reach the + # unattributable "The operation was canceled". + dest = tmp_path / "m.gguf" + clock = iter([0.0] + [1.0] * 8) # started, then one read inside the budget + + class _Resp: + headers = {"Content-Length": str(100 << 20)} + + def read(self, _n: int) -> bytes: + return b"\0" * (1 << 20) # never ends + + def __enter__(self): + return self + + def __exit__(self, *a: object) -> bool: + return False + + monkeypatch.setattr(smoke.urllib.request, "urlopen", lambda *a, **k: _Resp()) + monkeypatch.setattr( + smoke.time, "monotonic", lambda: next(clock, 10.0) + ) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke._download_with_progress( + "https://example.invalid/m.gguf", dest, budget_s=5.0 + ) + assert exc_info.value.failure_class == smoke.FailureClass.DOWNLOAD + assert exc_info.value.exit_code == 2 + assert "budget" in str(exc_info.value) + # Its own remedy, naming why the bound exists at all — not the generic + # "download failed for : SmokeFailure(...)" a re-wrap would give. + assert exc_info.value.remedy != smoke.REMEDIES[smoke.FailureClass.DOWNLOAD] + assert "job clock" in exc_info.value.remedy + # The partial file still goes, or the next run checksum-fails on it. + assert not dest.exists() + + def test_one_quiet_stretch_cannot_outlast_the_whole_budget(self) -> None: + # urllib's timeout is per socket operation, so a read timeout at or + # above the transfer budget means a single silent connection decides how + # long the download runs, and the budget never gets to. + assert smoke.DOWNLOAD_READ_TIMEOUT_S < smoke.DOWNLOAD_BUDGET_S + def test_progress_is_logged_sparsely_not_once_per_megabyte( self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture ) -> None: @@ -1105,6 +1152,24 @@ def test_the_documented_target_is_under_thirty_minutes(self) -> None: def test_the_default_matches_what_the_workflow_passes(self) -> None: assert _workflow_budget_s() == smoke.DEFAULT_WALL_CLOCK_BUDGET_S + def test_the_download_cannot_eat_the_margin_it_runs_in(self) -> None: + # The download is a separate step that runs before the smoke, so it + # spends the same margin the assertion above reserves for setup, and the + # harness's own Deadline never sees it. Its budget therefore has to fit + # inside that margin alongside the other pre-smoke steps, or a slow + # Hugging Face night hands the smoke a job clock with no room left and + # GitHub cancels the job mid-conversation — no class, no phase, no + # remedy, which is the one outcome this whole design rules out. + other_pre_smoke_steps_s = 120.0 # checkout, 3 pip installs, cache restore + save + assert ( + smoke.DOWNLOAD_BUDGET_S + other_pre_smoke_steps_s + <= PRE_SMOKE_JOB_MINUTES * 60 + ), ( + f"a {smoke.DOWNLOAD_BUDGET_S:.0f} s download budget plus the other " + f"pre-smoke steps exceeds the ~{PRE_SMOKE_JOB_MINUTES} min margin between " + "the harness deadline and the job timeout" + ) + def _workflow_steps() -> list: import yaml From 0e00075f1a4f59a2e8e3e853983431b11b3793de Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 09:49:31 -0400 Subject: [PATCH 52/58] Stop a mistyped digest from deleting the model it cannot describe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit verify_model_checksum normalises the expected digest's case and surrounding whitespace precisely so that a hand-pasted hash is not mistaken for drift — its docstring says why: a mismatch deletes a 2.5 GB file and reports that the pinned upstream file was replaced and that the check must not be relaxed, which is an expensive and thoroughly misleading verdict to hand someone who typed the argument wrong. A digest that is the wrong length, or not hex at all, is the same mistake and was getting exactly that treatment. Copying 64 hex characters out of a terminal by hand and losing one is the ordinary way to produce it, and the local repro in docs/real-model-smoke.md asks the reader to do precisely that copy. No file's hash can ever match a 63-character string, so the comparison is not a check; it just destroys the download and blames upstream. So the digest now has to be a SHA-256 digest before it is compared to one: 64 hex characters, checked after normalisation and before the file is hashed. A malformed one is `pipeline` (the class this harness uses for inputs it cannot act on), carries a remedy naming both places the value can come from, leaves the file where it was, and does not spend minutes hashing gigabytes against something that cannot match. Real drift is untouched: still exit 3, still deletes the bad bytes. Both copies of the doc say so, in the Checksum-drift section and next to `--verify-only`, which is the flag most likely to be pointed at a model the reader wants to keep. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 10 +++++ docs/real-model-smoke.md | 10 +++++ scripts/nightly-model-smoke.py | 35 ++++++++++++++++- tests/scripts/test_nightly_model_smoke.py | 38 +++++++++++++++++++ 4 files changed, 92 insertions(+), 1 deletion(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index e4450e94..19ebe636 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -375,6 +375,14 @@ Exit 3 means one of two things: Never "fix" a drift by updating the expected hash to whatever is on disk — that is exactly the check this job exists to perform. +What exit 3 does *not* mean is that the digest it was handed is unusable. The +expected digest is normalised (case, surrounding whitespace) and then checked to +be 64 hex characters before anything is hashed; one that is not — a value copied +one character short out of a terminal, the realistic mistake when running the +repro below by hand — is exit 5 with its own remedy, and the model file is left +exactly where it was. Nothing no file's hash could ever match is allowed to cost +a 2.5 GB download or to be reported as upstream drift. + --- ## Running it locally @@ -414,6 +422,8 @@ Useful extras: other checksum check here, a mismatch **deletes** the file (exit 3), so a re-run re-downloads rather than re-verifying the same bad bytes — do not reach for it as a read-only inspection of a 2.5 GB download you want to keep. + A `--model-sha256` that is not 64 hex characters is rejected as exit 5 before + the file is touched, so a mistyped digest costs you nothing. - `--models-dir ` — look for `.gguf` somewhere other than `~/.convsim/models/llm/`. - `--wall-clock-budget-s` — the self-imposed deadline (default 1200 s). diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index 978b83df..df5113d9 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -371,6 +371,14 @@ Exit 3 means one of two things: Never "fix" a drift by updating the expected hash to whatever is on disk — that is exactly the check this job exists to perform. +What exit 3 does *not* mean is that the digest it was handed is unusable. The +expected digest is normalised (case, surrounding whitespace) and then checked to +be 64 hex characters before anything is hashed; one that is not — a value copied +one character short out of a terminal, the realistic mistake when running the +repro below by hand — is exit 5 with its own remedy, and the model file is left +exactly where it was. Nothing no file's hash could ever match is allowed to cost +a 2.5 GB download or to be reported as upstream drift. + --- ## Running it locally @@ -410,6 +418,8 @@ Useful extras: other checksum check here, a mismatch **deletes** the file (exit 3), so a re-run re-downloads rather than re-verifying the same bad bytes — do not reach for it as a read-only inspection of a 2.5 GB download you want to keep. + A `--model-sha256` that is not 64 hex characters is rejected as exit 5 before + the file is touched, so a mistyped digest costs you nothing. - `--models-dir ` — look for `.gguf` somewhere other than `~/.convsim/models/llm/`. - `--wall-clock-budget-s` — the self-imposed deadline (default 1200 s). diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 5de627eb..021968d3 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -202,6 +202,12 @@ class exit meaning # still show a stalled transfer; 2381 hide the verdict they precede. PROGRESS_LOG_STEP_PCT = 10 +# A SHA-256 digest is 64 hex characters. Anything else is not a digest the +# on-disk bytes could ever match, so comparing against it is not a check — see +# verify_model_checksum for why that distinction is worth 2.5 GB. +SHA256_HEX_CHARS = 64 +_HEX_DIGITS = frozenset("0123456789abcdef") + # A debrief summary shorter than this is not a usable coaching summary. MIN_SUMMARY_CHARS = 20 @@ -731,14 +737,41 @@ def verify_model_checksum(path: Path, expected: str, *, delete_on_mismatch: bool file was replaced … do NOT relax this check", which is an expensive and thoroughly misleading verdict to hand someone who pasted an upper-case hash into the local repro in docs/real-model-smoke.md. + + For the same reason, ``expected`` has to *be* a SHA-256 digest before it is + compared to one. A digest short by a character — the ordinary outcome of + copying 64 hex characters out of a terminal by hand — can never match any + file's hash, so the comparison is not a check at all; but it fails exactly + like drift does, which is to say it deletes the 2.5 GB file and reports + that the pinned upstream file was replaced and must not be re-pinned + lightly. A malformed digest is the harness being handed an input it cannot + act on, which is ``pipeline``, and the file is left alone. """ + expected = expected.strip().lower() + if len(expected) != SHA256_HEX_CHARS or not set(expected) <= _HEX_DIGITS: + raise SmokeFailure( + FailureClass.PIPELINE, + f"Not a SHA-256 digest: {expected!r} ({len(expected)} characters, " + f"expected {SHA256_HEX_CHARS} hex). Nothing was verified and nothing " + "was deleted — no file's hash could match this.", + remedy=( + "The expected digest itself is malformed, so this says nothing " + "about the bytes on disk and the model file has been left in " + "place. If it came from --model-sha256 on the command line, " + "re-copy all 64 characters printed by " + "`--print-registry-model starter` (the repro block in " + "docs/real-model-smoke.md shows a placeholder, not a value). If " + "it came from the registry, repair download.sha256 in " + "model-registry/registry.yaml (re-pin with scripts/pin-model.py) " + "and validate with scripts/validate-registry.py." + ), + ) if not path.exists(): raise SmokeFailure( FailureClass.DOWNLOAD, f"Model file not found: {path}. The cache restore produced no model and no " "download ran — check the cache key and the download step.", ) - expected = expected.strip().lower() actual = sha256_file(path) if actual != expected: size_mb = path.stat().st_size // 1_048_576 diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index d1287813..9369f2b9 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -148,6 +148,44 @@ def test_drift_can_keep_the_file_for_inspection(self, tmp_path: Path) -> None: smoke.verify_model_checksum(path, "0" * 64, delete_on_mismatch=False) assert path.exists() + def test_a_malformed_digest_is_not_drift_and_keeps_the_file(self, tmp_path: Path) -> None: + # A digest short by one character is the ordinary outcome of copying 64 + # hex characters out of a terminal by hand, and it can never match any + # file's hash -- so comparing against it is not a check. Treating it as + # drift deleted a 2.5 GB download and reported that the pinned upstream + # file had been replaced, which is the expensive, misleading verdict the + # case/whitespace normalisation above exists to avoid. + path, digest = self._write(tmp_path) + for malformed in (digest[:-1], digest + "0", "not-a-digest", "", digest[:-1] + "g"): + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.verify_model_checksum(path, malformed) + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE, malformed + assert path.exists(), f"{malformed!r} must not delete the model" + # The checksum remedy ("the pinned upstream file was replaced ... + # do NOT relax this check") is the wrong advice here. + assert exc_info.value.remedy != smoke.REMEDIES[smoke.FailureClass.CHECKSUM] + + def test_a_malformed_digest_is_caught_before_the_file_is_hashed( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + # Hashing 2.5 GB to compare it against something that cannot match is + # pure waste, and a missing file would otherwise be reported as + # `download` while the real problem is the argument. + def _never(*args: object, **kwargs: object) -> str: + raise AssertionError("must not hash the file for a malformed digest") + + monkeypatch.setattr(smoke, "sha256_file", _never) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.verify_model_checksum(tmp_path / "absent.gguf", "deadbeef") + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + + def test_the_registry_pin_is_a_well_formed_digest(self) -> None: + # The one digest the nightly actually runs on has to pass the guard + # above, or the job fails before it fetches anything. + model = smoke.resolve_registry_model("starter") + assert len(model["sha256"]) == smoke.SHA256_HEX_CHARS + assert set(model["sha256"].lower()) <= smoke._HEX_DIGITS + def test_missing_file_is_a_download_failure_not_a_checksum_failure(self, tmp_path: Path) -> None: with pytest.raises(smoke.SmokeFailure) as exc_info: smoke.verify_model_checksum(tmp_path / "absent.gguf", "0" * 64) From 01f53c3bcc77756a38d718139b4f77e0eeab2ce7 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 10:40:10 -0400 Subject: [PATCH 53/58] Stop a mistyped digest from costing the download it cannot describe verify_model_checksum rejects a malformed --model-sha256 without hashing anything, but download_model reached it only after the transfer: a digest copied one character short out of a terminal bought a 2.5 GB fetch and then exit 5. The realistic source of that mistake is the local repro's hand-pasted --model-sha256, which runs --download-only, so the reader waited out the whole download to be told the argument was wrong -- and docs/real-model-smoke.md already promises the opposite ("Nothing no file's hash could ever match is allowed to cost a 2.5 GB download"). No bytes could have changed the verdict, so the shape guard moves into _require_sha256_digest and now runs before the transfer as well as before the hash. The failure text says so too: "Nothing was fetched" leads the message that already said nothing was verified and nothing was deleted. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 61 +++++++++++++++-------- tests/scripts/test_nightly_model_smoke.py | 18 +++++++ 2 files changed, 59 insertions(+), 20 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 021968d3..e3b7516c 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -723,6 +723,35 @@ def sha256_file(path: Path) -> str: return h.hexdigest() +def _require_sha256_digest(expected: str) -> str: + """Normalise ``expected`` and reject anything that is not a SHA-256 digest. + + Separate from ``verify_model_checksum`` so it can also run *before* a + transfer: the digest is an argument, and nothing about the 2.5 GB that + would be fetched can make a malformed one match. See ``download_model``. + """ + expected = expected.strip().lower() + if len(expected) != SHA256_HEX_CHARS or not set(expected) <= _HEX_DIGITS: + raise SmokeFailure( + FailureClass.PIPELINE, + f"Not a SHA-256 digest: {expected!r} ({len(expected)} characters, " + f"expected {SHA256_HEX_CHARS} hex). Nothing was fetched, nothing was " + "verified and nothing was deleted — no file's hash could match this.", + remedy=( + "The expected digest itself is malformed, so this says nothing " + "about the bytes on disk and the model file has been left in " + "place. If it came from --model-sha256 on the command line, " + "re-copy all 64 characters printed by " + "`--print-registry-model starter` (the repro block in " + "docs/real-model-smoke.md shows a placeholder, not a value). If " + "it came from the registry, repair download.sha256 in " + "model-registry/registry.yaml (re-pin with scripts/pin-model.py) " + "and validate with scripts/validate-registry.py." + ), + ) + return expected + + def verify_model_checksum(path: Path, expected: str, *, delete_on_mismatch: bool = True) -> str: """Hash ``path`` and compare to ``expected``; raise on drift. @@ -745,27 +774,11 @@ def verify_model_checksum(path: Path, expected: str, *, delete_on_mismatch: bool like drift does, which is to say it deletes the 2.5 GB file and reports that the pinned upstream file was replaced and must not be re-pinned lightly. A malformed digest is the harness being handed an input it cannot - act on, which is ``pipeline``, and the file is left alone. + act on, which is ``pipeline``, and the file is left alone. ``download_model`` + applies the same guard before it fetches anything, so the verdict does not + cost a transfer either. """ - expected = expected.strip().lower() - if len(expected) != SHA256_HEX_CHARS or not set(expected) <= _HEX_DIGITS: - raise SmokeFailure( - FailureClass.PIPELINE, - f"Not a SHA-256 digest: {expected!r} ({len(expected)} characters, " - f"expected {SHA256_HEX_CHARS} hex). Nothing was verified and nothing " - "was deleted — no file's hash could match this.", - remedy=( - "The expected digest itself is malformed, so this says nothing " - "about the bytes on disk and the model file has been left in " - "place. If it came from --model-sha256 on the command line, " - "re-copy all 64 characters printed by " - "`--print-registry-model starter` (the repro block in " - "docs/real-model-smoke.md shows a placeholder, not a value). If " - "it came from the registry, repair download.sha256 in " - "model-registry/registry.yaml (re-pin with scripts/pin-model.py) " - "and validate with scripts/validate-registry.py." - ), - ) + expected = _require_sha256_digest(expected) if not path.exists(): raise SmokeFailure( FailureClass.DOWNLOAD, @@ -788,6 +801,14 @@ def verify_model_checksum(path: Path, expected: str, *, delete_on_mismatch: bool def download_model(url: str, sha256: str, model_id: str, models_dir: Path = MODELS_DIR) -> Path: + # The digest is checked for *shape* before the transfer, not after it. + # verify_model_checksum below rejects a malformed digest without hashing + # anything, but reaching it means 2.5 GB has already been fetched to be + # compared against a value no file could ever match — and on the realistic + # route here, the local repro's hand-pasted --model-sha256, the reader waits + # out the whole download to be told the argument was wrong. There is no + # verdict the bytes could have changed, so spend nothing to reach it. + sha256 = _require_sha256_digest(sha256) dest = models_dir / f"{model_id}.gguf" if dest.exists(): print(f" Model already on disk: {dest}") diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 9369f2b9..03960d90 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -179,6 +179,24 @@ def _never(*args: object, **kwargs: object) -> str: smoke.verify_model_checksum(tmp_path / "absent.gguf", "deadbeef") assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + def test_a_malformed_digest_is_rejected_before_anything_is_fetched( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + # Rejecting it only at verification time means the reader waits out a + # 2.5 GB download to be told the argument was wrong -- and the realistic + # source of a malformed digest is exactly the hand-pasted --model-sha256 + # of the local repro, which runs --download-only. No bytes could have + # changed the verdict, and docs/real-model-smoke.md promises the download + # is not spent on one. + def _never(*args: object, **kwargs: object) -> None: + raise AssertionError("must not fetch 2.5 GB to compare it to a non-digest") + + monkeypatch.setattr(smoke, "_download_with_progress", _never) + with pytest.raises(smoke.SmokeFailure) as exc_info: + smoke.download_model("https://example.invalid/m.gguf", "deadbeef", "m", tmp_path) + assert exc_info.value.failure_class == smoke.FailureClass.PIPELINE + assert not (tmp_path / "m.gguf").exists() + def test_the_registry_pin_is_a_well_formed_digest(self) -> None: # The one digest the nightly actually runs on has to pass the guard # above, or the job fails before it fetches anything. From 525c08d81139aa76deebe0eecd69a51725a2de26 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 10:58:12 -0400 Subject: [PATCH 54/58] Warn about the turns the fallback check skipped, not just a missing payload MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `_debug_flags_by_turn` drops any /debug entry it cannot key by turn number, so the parse flags can cover some of the generated turns and not others. `evaluate_turns` only warned when *every* generated turn lacked them, which left the partial case silent — and not harmlessly: an absent `used_fallback` is falsy, so an unflagged turn counts towards "at least one generated turn was a real reply" with no evidence at all. With a payload that keyed turns 2 and 3 as fallbacks and dropped turn 1, the run went green reporting only "2 of 3 model-generated NPC turns fell back", with nothing anywhere in the report or step summary to say that the third turn — the one keeping the all-fell-back failure from firing — was never checked. The only trace was a print buried in the job log. The rubric path already refuses to round an unreadable turn down to zero (`_total_rubric_observations` returns None rather than 0 for exactly this situation); this is the same refusal applied to the fallback flags. All of them missing still reads as the whole check not running; some of them missing now names the count and the turns, and says the partial payload is itself worth a look at the turn response contract. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 38 ++++++++++--- tests/scripts/test_nightly_model_smoke.py | 66 +++++++++++++++++++++++ 2 files changed, 97 insertions(+), 7 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index e3b7516c..8421b8dd 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1152,9 +1152,12 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: fell back to the canned safe utterance then no real NPC turns were produced and the smoke proved nothing, so that is a failure. - Turns carrying no ``used_fallback`` flag at all — the debug endpoint is + Turns carrying no ``used_fallback`` flag — the debug endpoint is best-effort — warn that the check did not run, so an absent proof is not - mistaken for a passed one. + mistaken for a passed one. Counted per turn rather than all-or-nothing: a + payload that keys only some of the turns leaves the same hole on the ones it + missed, and an absent flag is falsy, so those turns would otherwise be + credited as real replies by the checks below without any evidence. ``replayed_opening`` gets the identical policy for the other way a turn can pass every flag without being a reply: reciting the authored opening back at @@ -1182,16 +1185,37 @@ def evaluate_turns(turns: List[Dict[str, Any]]) -> tuple[List[str], List[str]]: failures.append("No model-generated NPC turns were recorded") return (failures, warnings) - if not any("used_fallback" in t for t in generated): - # The flags come from the debug endpoint, which is best-effort. Without - # them a canned fallback is indistinguishable from real model output — - # it is a non-empty NPC utterance too — so the check below silently - # passes. Say so, rather than let the report imply a proof we do not have. + # The flags come from the debug endpoint, which is best-effort. Without + # them a canned fallback is indistinguishable from real model output — it is + # a non-empty NPC utterance too — so the checks below silently pass. Say so, + # rather than let the report imply a proof we do not have. + # + # Counted per turn, not all-or-nothing: _debug_flags_by_turn drops any entry + # it cannot key by turn number, so a changed /debug contract can cover some + # turns and not others. On the turns it missed the hole is exactly the same, + # and worse than harmless — an absent used_fallback is falsy, so those turns + # count towards "at least one generated turn was a real reply" in the checks + # below and can keep a conversation of fallbacks from failing at all. The + # rubric path already refuses to round an unreadable turn down to zero (see + # _total_rubric_observations); this is the same refusal. + unflagged = [t for t in generated if "used_fallback" not in t] + if len(unflagged) == len(generated): warnings.append( "Per-turn parse flags were unavailable, so the fallback check did not " "run: this run cannot prove the NPC turns were model output rather " "than the canned safe utterance" ) + elif unflagged: + missing = ", ".join( + str(t.get("label") or f"turn {t.get('turn_number')}") for t in unflagged + ) + warnings.append( + f"Per-turn parse flags were missing for {len(unflagged)} of " + f"{len(generated)} model-generated NPC turns ({missing}), so the " + "fallback check did not run for those: they count as real replies " + "below on no evidence at all. The /debug payload keyed only some of " + "the turns, which is itself worth a look at the turn response contract" + ) fallbacks = [t for t in generated if t.get("used_fallback")] all_fell_back = _all_generated_turns_fell_back(turns) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 03960d90..140ea164 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -610,11 +610,45 @@ def test_missing_parse_flags_warn_that_the_fallback_check_did_not_run(self) -> N assert failures == [] assert any("fallback check did not run" in w for w in warnings) + def test_parse_flags_covering_only_some_turns_warn_about_the_rest(self) -> None: + # _debug_flags_by_turn drops any /debug entry it cannot key by turn + # number, so the payload can cover some turns and not others. An absent + # used_fallback is falsy, which means the uncovered turns are credited + # as real replies with no evidence at all — here they are the only + # reason the run is not failed for a conversation of fallbacks. The + # report has to say which turns the check skipped. + turns = [ + {"label": "player_turn_1", "turn_number": 1, + "model_generated": True, "npc_excerpt": "Go on."}, + _turn(label="player_turn_2", turn_number=2, used_fallback=True), + _turn(label="player_turn_3", turn_number=3, used_fallback=True), + ] + failures, warnings = smoke.evaluate_turns(turns) + assert failures == [] + warning = next(w for w in warnings if "fallback check did not run" in w) + assert "1 of 3" in warning + assert "player_turn_1" in warning + + def test_an_unlabelled_turn_without_flags_is_named_by_its_number(self) -> None: + turns = [ + {"turn_number": 1, "model_generated": True, "npc_excerpt": "Go on."}, + _turn(turn_number=2), + ] + _, warnings = smoke.evaluate_turns(turns) + assert any("turn 1" in w for w in warnings) + def test_present_parse_flags_do_not_warn(self) -> None: failures, warnings = smoke.evaluate_turns([_turn()]) assert failures == [] assert warnings == [] + def test_fully_covered_parse_flags_do_not_warn_about_partial_coverage(self) -> None: + failures, warnings = smoke.evaluate_turns( + [_turn(), _turn(label="player_turn_2", turn_number=2)] + ) + assert failures == [] + assert not any("missing for" in w for w in warnings) + def test_opening_only_run_fails(self) -> None: turns = [{"label": "npc_opening", "turn_number": 0, "model_generated": False, "npc_excerpt": "Thanks for coming in."}] @@ -2827,6 +2861,38 @@ def _bad_debug(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> di assert results["verdict"] == "pass" assert any("fallback check did not run" in w for w in results["warnings"]) + def test_a_debug_payload_that_keys_only_some_turns_says_which_it_skipped( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # A partial payload is the same hole as a missing one, on the turns it + # missed — and here it is load-bearing: turn 1 having no flag is the + # only reason "every generated turn fell back" does not fire, so the + # run stays green on a conversation that may have had no real reply in + # it. The warning has to name the turn the check skipped rather than + # leave the reader with "2 of 3 fell back" and no hint about the third. + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core( + _debrief(), + debug_turns=[ + {"turn_number": 2, "used_fallback": True}, + {"turn_number": 3, "used_fallback": True}, + {"used_fallback": False}, # turn 1: unkeyable, so dropped + ], + )) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == 0 + results = json.loads(report.read_text(encoding="utf-8")) + warning = next( + w for w in results["warnings"] if "fallback check did not run" in w + ) + assert "1 of 3" in warning + assert "player_turn_1" in warning + # --------------------------------------------------------------------------- # CLI entry point From 9f657498758614111a49f2ef3cb31cf93d0f621d Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 11:21:34 -0400 Subject: [PATCH 55/58] Re-derive the budget's measurements from every nightly log GitHub still holds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runtime-budget table, the CI hardware factor and the recited-opening evidence were all sampled from ten nightlies described as "still in GitHub's log retention". Twelve are: the two oldest, 2026-09-22 and 2026-09-23, were left out, so a claim presented as the complete retained record was a subset of it. Re-read from the job API and the run logs for all twelve: - turn latency gains two samples (71 s and 94 s). The range 66-115 s is unchanged, so CI_HARDWARE_FACTOR 20 and the 240 s ceiling stand, but the median moves 108 -> 104 s and the slow end is ~11.5x the documented budget, not ~11.6x; - the recital check keeps its evidence: both added runs also logged an NPC reply whose leading characters were byte-identical to `opening_npc_says`, so the observation is twelve for twelve rather than ten for ten; - the per-step rows were measured against the step timings rather than estimated: set-up is ~5 s (not ~10 s), a warm pip install ~14 s, a cache restore ~24 s with a 19-28 s spread, and the one cold run's download ~10 s. The cache bullet also compared unlike things. "A cache miss (~9 s to fetch and hash) is cheaper than a cache hit (~25 s to restore)" dropped the ~10 s cache save that only a miss pays, inflating a real ~10 s gap into an apparent 16 s one; a reader tuning this would conclude the cache actively costs clock. It measures 31 s cold against 38-51 s warm, so the conclusion — the cache saves the network, not the clock, and abandoning a poisoned entry costs nothing — survives with the save counted. That measured 31-51 s pre-smoke total now replaces the rounded "~1 min" wherever the 9 min margin is justified against it. No behaviour change: docs, the workflow header and one harness docstring only. Co-Authored-By: Claude Opus 5 --- .github/workflows/model-smoke-nightly.yml | 52 ++++++++-------- .../src/content/docs/dev/real-model-smoke.md | 61 ++++++++++--------- docs/real-model-smoke.md | 61 ++++++++++--------- scripts/nightly-model-smoke.py | 16 ++--- 4 files changed, 101 insertions(+), 89 deletions(-) diff --git a/.github/workflows/model-smoke-nightly.yml b/.github/workflows/model-smoke-nightly.yml index 51fb3799..bd156e8a 100644 --- a/.github/workflows/model-smoke-nightly.yml +++ b/.github/workflows/model-smoke-nightly.yml @@ -38,17 +38,17 @@ # 10 min gap is deliberate margin, not an accounting of known work: the steps # before the smoke (checkout, pip install, cache restore and, on a cache miss, # the download plus the cache save) run against the same job clock and are -# invisible to the script's own deadline, but they measure ~1 min in total. +# invisible to the script's own deadline, but they measure 31–51 s in total. # The margin is sized for the network-bound part of that, which a slow # Hugging Face or Actions-cache night can stretch by minutes. The download is # the one of those steps that could stretch without limit, so it carries its own # 5 min budget (DOWNLOAD_BUDGET_S in the script) and fails as `download` rather # than spending the margin the smoke still needs. # -# set up job + checkout + setup-python ~10 s -# pip install ×3 (cached wheels) ~16 s cold, ~13 s warm -# model cache restore (2.3 GiB) ~25 s on a hit, <1 s on a miss -# download + SHA-256 verify (cache miss) ~9 s — bounded at 5 min +# set up job + checkout + setup-python ~5 s +# pip install ×3 (cached wheels) ~16 s cold, ~14 s warm +# model cache restore (2.3 GiB) ~24 s on a hit, <1 s on a miss +# download + SHA-256 verify (cache miss) ~10 s — bounded at 5 min # cache save (cache miss only) ~10 s # SHA-256 re-verify before load <10 s — bounded, see below # model load + core start + session ~8 s @@ -57,19 +57,20 @@ # ------------------------------------------------ # total ~11 min, warm cache or cold # -# The non-inference figures are measured, from this job's own logs for the ten -# nightlies to 2026-10-03. Two of them are counter-intuitive and worth knowing -# before tuning anything: a cache MISS (~9 s to fetch and hash the GGUF) is -# cheaper than a cache HIT (~25 s to restore the same bytes), so the cache -# saves the network and not the clock; and everything before the conversation -# is ~1 min of a 30 min job, so the budget is inference and nothing else. +# The non-inference figures are measured, from this job's own logs for the +# twelve nightlies whose logs GitHub still held on 2026-10-03. Two of them are +# counter-intuitive and worth knowing before tuning anything: a cache MISS is no +# more expensive than a cache HIT — ~10 s to fetch and hash the GGUF plus ~10 s +# to save it, against ~19–28 s to restore the same bytes — so the cache saves +# the network and not the clock; and everything before the conversation is +# 31–51 s of a 30 min job, so the budget is inference and nothing else. # The re-verify row is the one with no direct measurement — the previous # single-turn harness did not re-verify — and is bounded by the cold row above, -# where downloading 2.3 GiB *and* hashing it together took 9 s. +# where downloading 2.3 GiB *and* hashing it together took 10 s. # # The inference figures are projected from the one latency this job has -# actually measured — a single behavioral-interview turn, ten nightlies: -# 66/93/98/103/104/111/112/114/115/115 s — rounded up to ~120 s per turn +# actually measured — a single behavioral-interview turn, twelve nightlies: +# 66/71/93/94/98/103/104/111/112/114/115/115 s — rounded up to ~120 s per turn # because turns 2 and 3 render a longer transcript than turn 1, with the # debrief allowed twice a turn. Replace both with real per-phase durations # from the first green nightly's report artifact. @@ -85,17 +86,18 @@ # CI budget = documented budget × CI_HARDWARE_FACTOR (currently 20) # # The factor is calibrated empirically: a full behavioral-interview turn on -# this runner has measured 66–115 s across the ten nightlies to 2026-10-03 -# (median ~108 s — large grammar-constrained prompt, 4B-Q4 model, CPU-only), -# i.e. ~11.6× the 10 s documented budget at the slow end, which is the end the -# factor is set from. 20× gives a 10 s × 20 = 200 s CI budget and, with -# the script's 1.20 regression tolerance, a 240 s ceiling — roughly 2× headroom -# over the observed latency so normal runner-to-runner variance does not flap -# the nightly, while a genuine >2× regression still fails. The headline -# full_response_ms is the MEDIAN of the scripted turns, so one unlucky turn -# cannot flap the job. This factor only models CI-hardware slowness; the -# product's 10 s target-hardware SLO is unchanged. Re-measure and re-tune if -# the runner class or starter model changes. +# this runner has measured 66–115 s across the twelve nightlies whose logs +# GitHub still held on 2026-10-03 (median ~104 s — large grammar-constrained +# prompt, 4B-Q4 model, CPU-only), i.e. ~11.5× the 10 s documented budget at the +# slow end, which is the end the factor is set from. 20× gives a +# 10 s × 20 = 200 s CI budget and, with the script's 1.20 regression tolerance, +# a 240 s ceiling — roughly 2× headroom over the observed latency so normal +# runner-to-runner variance does not flap the nightly, while a genuine >2× +# regression still fails. The headline full_response_ms is the MEDIAN of the +# scripted turns, so one unlucky turn cannot flap the job. This factor only +# models CI-hardware slowness; the product's 10 s target-hardware SLO is +# unchanged. Re-measure and re-tune if the runner class or starter model +# changes. name: Model smoke nightly diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 19ebe636..6bafd9c5 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -57,9 +57,9 @@ The NPC *opening* line is authored scenario text, not a generation — so the scripted player turns, not the opening, are what prove the model is working. A turn that is real model output is not automatically a *reply*, and the -per-turn parse flags cannot tell the difference. All ten nightly runs of the -previous single-turn harness still in GitHub's log retention logged an NPC -answer whose leading characters were byte-identical to the scenario's +per-turn parse flags cannot tell the difference. All twelve nightly runs of the +previous single-turn harness whose logs GitHub still held on 2026-10-03 logged +an NPC answer whose leading characters were byte-identical to the scenario's `opening_npc_says` — the model reciting the opening question back instead of answering it. `used_fallback` stays false for those turns (the utterance is neither empty nor the canned safe one), so the run went green on a conversation @@ -68,9 +68,9 @@ and applies the same policy it applies to fallbacks: one such turn warns, *every* generated turn doing it fails, because then the model answered nothing and the run proves nothing. -Read that ten-for-ten with its cause attached, because it is not evidence that -the starter model always recites. The single player turn those runs submitted -was `"Reply with exactly one sentence: Hello, I am ready."` — an instruction +Read that twelve-for-twelve with its cause attached, because it is not evidence +that the starter model always recites. The single player turn those runs +submitted was `"Reply with exactly one sentence: Hello, I am ready."` — an instruction addressed to the model rather than something an interview candidate says — and this job replaces it with the three in-character turns in `SCRIPTED_PLAYER_TURNS`, which the real starter model answered properly in local @@ -255,7 +255,7 @@ The 10-minute gap between the two is deliberate margin, not an accounting of known work. `timeout-minutes` covers the whole job, and the harness only starts after checkout, three `pip install` steps, the model cache restore and — on a cache miss — the download and the cache save that follows it, which together -measure **~1 min** on the nightlies run so far, not the several +measure **31–51 s** on the nightlies run so far, not the several minutes a 2.5 GB transfer sounds like. The gap is sized for the part of that work that is network-bound rather than for its measured cost: a slow Hugging Face or Actions-cache night can cost minutes that the harness's own @@ -266,10 +266,10 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | |---|---|---| -| Set up job + checkout + `setup-python` | ~10 s | ~10 s | -| `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~13 s | -| Model cache restore (2.3 GiB) | — (miss: <1 s) | ~25 s | -| Model download from Hugging Face + SHA-256 verify (2.3 GiB) ◊ | ~9 s | — | +| Set up job + checkout + `setup-python` | ~5 s | ~5 s | +| `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~14 s | +| Model cache restore (2.3 GiB) | — (miss: <1 s) | ~24 s | +| Model download from Hugging Face + SHA-256 verify (2.3 GiB) ◊ | ~10 s | — | | Cache save (2.3 GiB, its own step before the smoke) | ~10 s | — | | SHA-256 re-verify before the weights are loaded ‡ | <10 s | <10 s | | `llama-server` model load + `convsim-core` start + session create | ~8 s | ~8 s | @@ -277,16 +277,19 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Debrief generation † | ~4 min | ~4 min | | **Total** | **~11 min** | **~11 min** | -The non-inference rows are measured, from the job's own logs for the ten -nightlies to 2026-10-03 — the cold-cache rows from the one of those that missed -the cache. Two things in them are worth knowing before tuning anything: +The non-inference rows are measured, from the job's own logs for the twelve +nightlies whose logs GitHub still held on 2026-10-03 — the cold-cache rows from +the one of those that missed the cache. Two things in them are worth knowing +before tuning anything: -- **A cache miss is cheaper than a cache hit.** Downloading the GGUF from - Hugging Face and hashing it takes ~9 s; restoring the same bytes from the - Actions cache takes ~25 s. Caching the model saves the *network*, not the - clock, and abandoning a poisoned cache entry costs nothing. -- **Everything before the conversation is noise.** All of it together is ~1 min - of a 30 min job. The budget is inference, and only inference. +- **A cache miss costs no more than a cache hit.** Fetching the GGUF from + Hugging Face and hashing it takes ~10 s, and saving it to the Actions cache + another ~10 s; restoring the same bytes on a hit takes 19–28 s. Caching the + model saves the *network*, not the clock, and abandoning a poisoned cache + entry costs nothing. +- **Everything before the conversation is noise.** All of it together measured + 31–51 s across those twelve runs, in a 30 min job. The budget is inference, + and only inference. ◊ The download is the one pre-smoke step that could stretch without limit, and it runs where the harness's own deadline cannot see it, so it carries its own @@ -296,18 +299,19 @@ transfer, so a connection that goes quiet repeatedly — or merely crawls — wo otherwise spend the whole margin above and then hand the smoke a job clock with no room in it, which is the one way left to reach an unattributable cancellation. Exceeding either bound is exit 2 with its own remedy. 5 min is -~33× the measured 9 s, so a transfer has to be pathologically slow to hit it. +~30× the measured 10 s, so a transfer has to be pathologically slow to hit it. ‡ The harness re-verifies the on-disk file at the start of every run, which the previous single-turn harness did not, so this row has no direct measurement. The bound comes from the cold-cache row above: downloading 2.3 GiB *and* -hashing it took 9 s in total, so the hash alone is a few seconds. +hashing it took 10 s in total, so the hash alone is a few seconds. † The two inference rows are *projected*, because the multi-turn conversation and the debrief have never run on a runner. The projection comes from the only -latency this job has measured — one behavioral-interview turn, ten nightlies: -66 / 93 / 98 / 103 / 104 / 111 / 112 / 114 / 115 / 115 s — rounded up to ~120 s -per turn, since turns 2 and 3 render a longer transcript than turn 1, and a +latency this job has measured — one behavioral-interview turn, twelve +nightlies: 66 / 71 / 93 / 94 / 98 / 103 / 104 / 111 / 112 / 114 / 115 / 115 s — +rounded up to ~120 s per turn, since turns 2 and 3 render a longer transcript +than turn 1, and a debrief allowed twice a turn (`DEBRIEF_SLOWDOWN_FACTOR`). Replace both rows with the real `phase_durations_s` from the first green nightly's report artifact. @@ -336,9 +340,10 @@ CI ceiling = documented budget × CI_HARDWARE_FACTOR × REGRESSION_TOLERANCE ``` `CI_HARDWARE_FACTOR` (20) is calibrated empirically: a full behavioral-interview -turn on this runner has measured 66–115 s across the ten nightlies to -2026-10-03 (median ~108 s), i.e. ~11.6× the 10 s documented budget at the slow -end. The factor is set from that slow end, not the median, and still leaves +turn on this runner has measured 66–115 s across the twelve nightlies whose +logs GitHub still held on 2026-10-03 (median ~104 s), i.e. ~11.5× the 10 s +documented budget at the slow end. The factor is set from that slow end, not the +median, and still leaves roughly 2× headroom — note the 66 s run against the 115 s one, which is the night-to-night spread a single-sample calibration would have missed. A genuine >2× regression still fails. The headline `full_response_ms` is the **median** of diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index df5113d9..d5e0e5d8 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -53,9 +53,9 @@ The NPC *opening* line is authored scenario text, not a generation — so the scripted player turns, not the opening, are what prove the model is working. A turn that is real model output is not automatically a *reply*, and the -per-turn parse flags cannot tell the difference. All ten nightly runs of the -previous single-turn harness still in GitHub's log retention logged an NPC -answer whose leading characters were byte-identical to the scenario's +per-turn parse flags cannot tell the difference. All twelve nightly runs of the +previous single-turn harness whose logs GitHub still held on 2026-10-03 logged +an NPC answer whose leading characters were byte-identical to the scenario's `opening_npc_says` — the model reciting the opening question back instead of answering it. `used_fallback` stays false for those turns (the utterance is neither empty nor the canned safe one), so the run went green on a conversation @@ -64,9 +64,9 @@ and applies the same policy it applies to fallbacks: one such turn warns, *every* generated turn doing it fails, because then the model answered nothing and the run proves nothing. -Read that ten-for-ten with its cause attached, because it is not evidence that -the starter model always recites. The single player turn those runs submitted -was `"Reply with exactly one sentence: Hello, I am ready."` — an instruction +Read that twelve-for-twelve with its cause attached, because it is not evidence +that the starter model always recites. The single player turn those runs +submitted was `"Reply with exactly one sentence: Hello, I am ready."` — an instruction addressed to the model rather than something an interview candidate says — and this job replaces it with the three in-character turns in `SCRIPTED_PLAYER_TURNS`, which the real starter model answered properly in local @@ -251,7 +251,7 @@ The 10-minute gap between the two is deliberate margin, not an accounting of known work. `timeout-minutes` covers the whole job, and the harness only starts after checkout, three `pip install` steps, the model cache restore and — on a cache miss — the download and the cache save that follows it, which together -measure **~1 min** on the nightlies run so far, not the several +measure **31–51 s** on the nightlies run so far, not the several minutes a 2.5 GB transfer sounds like. The gap is sized for the part of that work that is network-bound rather than for its measured cost: a slow Hugging Face or Actions-cache night can cost minutes that the harness's own @@ -262,10 +262,10 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Phase | Cold cache | Warm cache | |---|---|---| -| Set up job + checkout + `setup-python` | ~10 s | ~10 s | -| `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~13 s | -| Model cache restore (2.3 GiB) | — (miss: <1 s) | ~25 s | -| Model download from Hugging Face + SHA-256 verify (2.3 GiB) ◊ | ~9 s | — | +| Set up job + checkout + `setup-python` | ~5 s | ~5 s | +| `pip install` ×3 (prompt-composer, convsim-core, llama-cpp-python wheel) | ~16 s | ~14 s | +| Model cache restore (2.3 GiB) | — (miss: <1 s) | ~24 s | +| Model download from Hugging Face + SHA-256 verify (2.3 GiB) ◊ | ~10 s | — | | Cache save (2.3 GiB, its own step before the smoke) | ~10 s | — | | SHA-256 re-verify before the weights are loaded ‡ | <10 s | <10 s | | `llama-server` model load + `convsim-core` start + session create | ~8 s | ~8 s | @@ -273,16 +273,19 @@ On `ubuntu-latest` (CPU-only, 4B Q4\_K\_M): | Debrief generation † | ~4 min | ~4 min | | **Total** | **~11 min** | **~11 min** | -The non-inference rows are measured, from the job's own logs for the ten -nightlies to 2026-10-03 — the cold-cache rows from the one of those that missed -the cache. Two things in them are worth knowing before tuning anything: +The non-inference rows are measured, from the job's own logs for the twelve +nightlies whose logs GitHub still held on 2026-10-03 — the cold-cache rows from +the one of those that missed the cache. Two things in them are worth knowing +before tuning anything: -- **A cache miss is cheaper than a cache hit.** Downloading the GGUF from - Hugging Face and hashing it takes ~9 s; restoring the same bytes from the - Actions cache takes ~25 s. Caching the model saves the *network*, not the - clock, and abandoning a poisoned cache entry costs nothing. -- **Everything before the conversation is noise.** All of it together is ~1 min - of a 30 min job. The budget is inference, and only inference. +- **A cache miss costs no more than a cache hit.** Fetching the GGUF from + Hugging Face and hashing it takes ~10 s, and saving it to the Actions cache + another ~10 s; restoring the same bytes on a hit takes 19–28 s. Caching the + model saves the *network*, not the clock, and abandoning a poisoned cache + entry costs nothing. +- **Everything before the conversation is noise.** All of it together measured + 31–51 s across those twelve runs, in a 30 min job. The budget is inference, + and only inference. ◊ The download is the one pre-smoke step that could stretch without limit, and it runs where the harness's own deadline cannot see it, so it carries its own @@ -292,18 +295,19 @@ transfer, so a connection that goes quiet repeatedly — or merely crawls — wo otherwise spend the whole margin above and then hand the smoke a job clock with no room in it, which is the one way left to reach an unattributable cancellation. Exceeding either bound is exit 2 with its own remedy. 5 min is -~33× the measured 9 s, so a transfer has to be pathologically slow to hit it. +~30× the measured 10 s, so a transfer has to be pathologically slow to hit it. ‡ The harness re-verifies the on-disk file at the start of every run, which the previous single-turn harness did not, so this row has no direct measurement. The bound comes from the cold-cache row above: downloading 2.3 GiB *and* -hashing it took 9 s in total, so the hash alone is a few seconds. +hashing it took 10 s in total, so the hash alone is a few seconds. † The two inference rows are *projected*, because the multi-turn conversation and the debrief have never run on a runner. The projection comes from the only -latency this job has measured — one behavioral-interview turn, ten nightlies: -66 / 93 / 98 / 103 / 104 / 111 / 112 / 114 / 115 / 115 s — rounded up to ~120 s -per turn, since turns 2 and 3 render a longer transcript than turn 1, and a +latency this job has measured — one behavioral-interview turn, twelve +nightlies: 66 / 71 / 93 / 94 / 98 / 103 / 104 / 111 / 112 / 114 / 115 / 115 s — +rounded up to ~120 s per turn, since turns 2 and 3 render a longer transcript +than turn 1, and a debrief allowed twice a turn (`DEBRIEF_SLOWDOWN_FACTOR`). Replace both rows with the real `phase_durations_s` from the first green nightly's report artifact. @@ -332,9 +336,10 @@ CI ceiling = documented budget × CI_HARDWARE_FACTOR × REGRESSION_TOLERANCE ``` `CI_HARDWARE_FACTOR` (20) is calibrated empirically: a full behavioral-interview -turn on this runner has measured 66–115 s across the ten nightlies to -2026-10-03 (median ~108 s), i.e. ~11.6× the 10 s documented budget at the slow -end. The factor is set from that slow end, not the median, and still leaves +turn on this runner has measured 66–115 s across the twelve nightlies whose +logs GitHub still held on 2026-10-03 (median ~104 s), i.e. ~11.5× the 10 s +documented budget at the slow end. The factor is set from that slow end, not the +median, and still leaves roughly 2× headroom — note the 66 s run against the 115 s one, which is the night-to-night spread a single-sample calibration would have missed. A genuine >2× regression still fails. The headline `full_response_ms` is the **median** of diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 8421b8dd..86a9ea5c 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1086,18 +1086,18 @@ def _replays_opening(npc_text: str, opening_text: str) -> bool: """True when an NPC reply opens by reciting the authored opening verbatim. The scenario's opening line sits in the transcript the turn prompt renders, - and the starter model can answer a player turn by copying it back: all ten - nightly runs of the previous single-turn harness still in GitHub's log - retention logged an NPC reply whose leading characters matched - ``opening_npc_says`` exactly. That is not a reply, but nothing downstream - notices — the utterance is non-empty and it is not + and the starter model can answer a player turn by copying it back: all + twelve nightly runs of the previous single-turn harness whose logs GitHub + still held on 2026-10-03 logged an NPC reply whose leading characters + matched ``opening_npc_says`` exactly. That is not a reply, but nothing + downstream notices — the utterance is non-empty and it is not ``SAFE_FALLBACK_UTTERANCE``, so ``used_fallback`` stays false and the per-turn parse flags report a healthy turn. Proving the turns are real therefore needs this check as well as those flags. - Those ten runs are evidence that the failure mode exists, not that it is the - norm: the player turn they submitted was "Reply with exactly one sentence: - Hello, I am ready." — an instruction to the model, not something an + Those twelve runs are evidence that the failure mode exists, not that it is + the norm: the player turn they submitted was "Reply with exactly one + sentence: Hello, I am ready." — an instruction to the model, not something an interview candidate says — and this harness replaces it with the in-character ``SCRIPTED_PLAYER_TURNS``, which the real starter model answered properly in local verification. So the expected rate here is zero From fb9136ec1c21e25e41c51a00acdf7d69e5ef460d Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 11:25:35 -0400 Subject: [PATCH 56/58] Test the harness's child-process wiring, which no other test executes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The orchestration tests fake out both children, so `_start_llama_server` and `_start_core` — the code that decides *what the nightly actually runs* — were the one part of the harness no test reached. Two of the env overrides they build fail silently when misspelled, because pydantic-settings ignores an unknown name and keeps the product default: - CONVSIM_RUNTIME_ID selects the real runtime over the deterministic fake. This one would at least be caught: the run checks /health afterwards. - CONVSIM_LLAMA_CPP_CHAT_TIMEOUT would not. Without it the adapter keeps its 180 s default, so the first turn slower than that is killed adapter-side and convsim-core answers 500 — class `runtime`, "the server crashed" — which is exactly the misclassification the override exists to prevent, on a night when the only thing wrong was slow inference. So the new tests assert the argv and the env, and then check every CONVSIM_* override the harness sets against the field names declared on ServiceConfig and LlamaCppConfig in the product source, which also catches a future override added under a name nothing reads. The data-directory assertion pins the privacy claim in docs/real-model-smoke.md: no database, log or pack outside the tree the finally block deletes. Also covered, all previously exercised only by hand: - `--download-only` succeeding — the workflow's own download step and step 3 of the documented local repro; - `--verify-only` accepting a matching file, and deleting a drifted one with exit 3, which is the behaviour the doc warns about; - a missing --model-id and a `--verify-only` without a digest exiting 5 rather than argparse's 2, which EXIT_CODES reads as `download`; - warnings reaching the step summary on a pass, since every warning the harness raises says what an otherwise-green verdict does not prove. Harness coverage 97% -> 99%; no production or harness behaviour changed. Co-Authored-By: Claude Opus 5 --- tests/scripts/test_nightly_model_smoke.py | 208 ++++++++++++++++++++++ 1 file changed, 208 insertions(+) diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 140ea164..6db98190 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -1469,6 +1469,20 @@ def test_pass_summary_tabulates_measurements_with_ceilings(self) -> None: assert "240000 ms" in text # 10 000 × 20 × 1.2 assert "| `debrief_ms` | 200000 ms | — |" in text + def test_a_green_run_still_surfaces_what_it_did_not_prove(self) -> None: + # Every warning the harness raises says what an otherwise-green verdict + # does not establish -- a fallback narrative, a recited opening, a + # conversation the NPC cut short, parse flags it could not read. The step + # summary is where they are read, so a pass that drops them reports more + # confidence than the run earned. + text = smoke.render_step_summary({ + "verdict": "pass", + "model_id": "qwen3-4b", + "warnings": ["Debrief narrative used the deterministic fallback"], + }) + assert "### Warnings" in text + assert "- Debrief narrative used the deterministic fallback" in text + def test_a_pre_run_failure_does_not_claim_a_zero_second_run(self) -> None: # --download-only / --verify-only and the registry lookup write a summary # without ever starting a clock; reporting "Wall clock: 0 s (budget 0 s)" @@ -3034,6 +3048,200 @@ def test_a_usage_error_is_not_mistaken_for_a_download_failure(self) -> None: assert exc_info.value.code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] assert exc_info.value.code != smoke.EXIT_CODES[smoke.FailureClass.DOWNLOAD] + def test_a_missing_model_id_is_a_classified_usage_error(self) -> None: + with pytest.raises(SystemExit) as exc_info: + smoke.main([]) + assert exc_info.value.code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + + def test_verify_only_without_a_digest_is_a_classified_usage_error(self) -> None: + with pytest.raises(SystemExit) as exc_info: + smoke.main(["--verify-only", "--model-id", "m"]) + assert exc_info.value.code == smoke.EXIT_CODES[smoke.FailureClass.PIPELINE] + + def test_download_only_verifies_what_it_fetched_and_exits_zero( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + # The workflow's own download step, and step 3 of the documented local + # repro. Only its failure modes were covered above, so a typo in the + # success path would break the nightly before the smoke ever ran. + models_dir = tmp_path / "models" + payload = b"\x00gguf stand-in" + + def _fake_fetch(url: str, dest: Path, *, budget_s: float = 0.0) -> None: + dest.parent.mkdir(parents=True, exist_ok=True) + dest.write_bytes(payload) + + monkeypatch.setattr(smoke, "_download_with_progress", _fake_fetch) + + exit_code = smoke.main([ + "--download-only", "--model-id", "m", + "--model-url", "https://example.invalid/m.gguf", + "--model-sha256", hashlib.sha256(payload).hexdigest(), + "--models-dir", str(models_dir), + ]) + + assert exit_code == 0 + assert (models_dir / "m.gguf").read_bytes() == payload + + def test_verify_only_accepts_a_matching_file_and_exits_zero( + self, staged_model + ) -> None: + models_dir, model_id, digest = staged_model + exit_code = smoke.main([ + "--verify-only", "--model-id", model_id, + "--model-sha256", digest, "--models-dir", str(models_dir), + ]) + assert exit_code == 0 + assert (models_dir / f"{model_id}.gguf").exists() + + def test_verify_only_on_drift_exits_three_and_deletes_the_file( + self, staged_model, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + # The documented warning about --verify-only: a mismatch deletes the + # file so a re-run re-downloads instead of re-verifying the same bytes. + models_dir, model_id, _ = staged_model + summary = tmp_path / "summary.md" + monkeypatch.setenv("GITHUB_STEP_SUMMARY", str(summary)) + + exit_code = smoke.main([ + "--verify-only", "--model-id", model_id, + "--model-sha256", "a" * 64, "--models-dir", str(models_dir), + ]) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.CHECKSUM] + assert not (models_dir / f"{model_id}.gguf").exists() + assert "`checksum`" in summary.read_text(encoding="utf-8") + + +# --------------------------------------------------------------------------- +# Child process wiring +# --------------------------------------------------------------------------- + + +def _annotated_settings_fields(path: Path, class_name: str) -> set: + """Field names declared on a pydantic-settings class in ``path``.""" + import re + + src = path.read_text(encoding="utf-8") + body = src[src.index(f"class {class_name}("):] + end = re.search(r"\n(?:@|class |def )", body[1:]) + if end: + body = body[: end.start() + 1] + return set(re.findall(r"^ ([a-z_][a-z0-9_]*)\s*:", body, re.M)) + + +class TestChildProcessWiring: + """The argv and the env that make the smoke a *real-model* smoke. + + Every orchestration test above fakes both children out, so the code that + builds this argv and this env is the one part of the harness no test runs — + and two of the values in it fail silently: + + * ``CONVSIM_RUNTIME_ID`` is the only thing selecting the real runtime over + the deterministic fake default. The run does check ``/health`` afterwards, + so this one at least fails loudly. + * ``CONVSIM_LLAMA_CPP_CHAT_TIMEOUT`` does not. Misspell it and the adapter + silently keeps its 180 s default, so the first turn slower than that is + killed adapter-side and convsim-core answers 500 — class ``runtime``, "the + server crashed" — which is precisely the misclassification the override + exists to prevent, on a night when the only thing wrong was slow + inference. + + Pydantic reads these from a prefix plus a field name, so a name that is not + a declared setting is silently ignored rather than rejected. The last test + therefore checks every override the harness sets against the product's own + settings classes. + """ + + _CORE_SRC = REPO_ROOT / "services" / "convsim-core" / "convsim_core" + + @pytest.fixture() + def spawned(self, monkeypatch: pytest.MonkeyPatch) -> dict: + captured: dict = {} + + def _popen(argv, **kwargs): + captured["argv"] = argv + captured["env"] = kwargs.get("env") + return _FakeProc() + + monkeypatch.setattr(smoke.subprocess, "Popen", _popen) + return captured + + def test_llama_server_is_started_on_the_verified_model_and_port( + self, spawned: dict, tmp_path: Path + ) -> None: + model = tmp_path / "m.gguf" + smoke._start_llama_server(model, 7356) + argv = spawned["argv"] + assert argv[1:3] == ["-m", "llama_cpp.server"] + assert argv[argv.index("--model") + 1] == str(model) + assert argv[argv.index("--port") + 1] == "7356" + assert argv[argv.index("--host") + 1] == "127.0.0.1" + + def test_core_is_wired_to_the_real_runtime_not_the_fake_default( + self, spawned: dict, tmp_path: Path + ) -> None: + smoke._start_core(tmp_path / "data", 7399, 7356, 540.0) + env = spawned["env"] + assert env["CONVSIM_RUNTIME_ID"] == "llama_cpp" + assert env["CONVSIM_LLAMA_CPP_BASE_URL"] == "http://127.0.0.1:7356" + + def test_the_adapter_outlasts_the_latency_the_run_means_to_measure( + self, spawned: dict, tmp_path: Path + ) -> None: + smoke._start_core(tmp_path / "data", 7399, 7356, 540.0) + env = spawned["env"] + # Both, not just the chat budget: TIMEOUT covers the model listing a + # cold runner does first, and a listing cut short is also a 500. + assert env["CONVSIM_LLAMA_CPP_CHAT_TIMEOUT"] == "540" + assert env["CONVSIM_LLAMA_CPP_TIMEOUT"] == "540" + + def test_core_is_confined_to_the_throwaway_data_directory( + self, spawned: dict, tmp_path: Path + ) -> None: + # docs/real-model-smoke.md's privacy claim: the run leaves no database, + # log or pack behind outside a directory the finally block deletes. + data_dir = tmp_path / "data" + smoke._start_core(data_dir, 7399, 7356, 540.0) + env = spawned["env"] + for key in ("CONVSIM_DATA_DIR", "CONVSIM_LOG_DIR", + "CONVSIM_DB_DIR", "CONVSIM_PACKS_DIR"): + assert env[key].startswith(str(data_dir)), key + + def test_every_override_names_a_setting_the_product_actually_reads( + self, spawned: dict, tmp_path: Path + ) -> None: + import os + + smoke._start_core(tmp_path / "data", 7399, 7356, 540.0) + env = spawned["env"] + overrides = sorted( + k for k, v in env.items() + if k.startswith("CONVSIM_") and os.environ.get(k) != v + ) + assert overrides, "the env the harness builds sets no CONVSIM_* override" + + pools = { + "CONVSIM_LLAMA_CPP_": ( + "LlamaCppConfig", + _annotated_settings_fields( + self._CORE_SRC / "runtime" / "llama_cpp.py", "LlamaCppConfig" + ), + ), + "CONVSIM_": ( + "ServiceConfig", + _annotated_settings_fields(self._CORE_SRC / "config.py", "ServiceConfig"), + ), + } + for name in overrides: + prefix = "CONVSIM_LLAMA_CPP_" if name.startswith("CONVSIM_LLAMA_CPP_") else "CONVSIM_" + class_name, fields = pools[prefix] + field = name[len(prefix):].lower() + assert field in fields, ( + f"{name} is not a {class_name} setting, so pydantic ignores it " + "and the harness silently runs on the product's default" + ) + # --------------------------------------------------------------------------- # Child stderr tails From b108cebf1b4a76e8f848f1c8a5a3fcd4d26fa85e Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 11:44:21 -0400 Subject: [PATCH 57/58] Stop a cut-off generation from reading as a crashed server MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every transport-level failure is a `runtime` failure, so a *live* convsim-core that answered 5xx lands in the same class as a dead one — and `runtime`'s class remedy opens by telling the reader to read a child stderr tail, then names runner OOM and an incompatible llama-cpp-python wheel as the likely causes. When no child exited, that tail shows a healthy server and none of those causes applies. The reachable case is the one `_start_core` already half-mitigates. convsim-core answers 504 TURN_TIMEOUT when the adapter's generation budget runs out mid-reply, and the harness deliberately sets that budget above the latency ceiling so a slow turn is measured and reported as `budget` rather than killed and misreported as a crash. But that moves the cliff rather than removing it: at --ci-hardware-factor 20 the budget is 540 s against a 1200 s wall clock, so a turn past it is still cut off with the clock unspent, and neither `timeout` nor `budget` takes the class over. Exit 4 was then honest about the transport and wrong about the cause — the same misclassification the raised adapter timeout exists to prevent, one cliff further out. So when a `runtime` verdict finds both children still alive, print a remedy that says so: nothing crashed, a 504 means inference roughly 5× slower than any nightly has measured, start from measured_ms and phase_durations_s rather than an OOM — and otherwise read the tails for a server that is up but wedged. Only when the failure was using the class default, so the port-conflict failure keeps its own advice, which is right about a run where no child was ever started. The doc's failure table sent the reader to a stderr tail unconditionally; it now distinguishes the three cases and links a new section carrying the reasoning, with tests pinning that the link resolves and that the adapter budget quoted in both the doc and the remedy is the one the harness computes. Co-Authored-By: Claude Opus 5 --- .../src/content/docs/dev/real-model-smoke.md | 28 ++++- docs/real-model-smoke.md | 28 ++++- scripts/nightly-model-smoke.py | 51 ++++++++ tests/scripts/test_nightly_model_smoke.py | 114 ++++++++++++++++++ 4 files changed, 219 insertions(+), 2 deletions(-) diff --git a/docs-site/src/content/docs/dev/real-model-smoke.md b/docs-site/src/content/docs/dev/real-model-smoke.md index 6bafd9c5..ca99fa74 100644 --- a/docs-site/src/content/docs/dev/real-model-smoke.md +++ b/docs-site/src/content/docs/dev/real-model-smoke.md @@ -118,7 +118,7 @@ log, and a remedy printed next to it: | 1 | `budget` | Pipeline worked; latency regressed past the CI ceiling | Compare `measured_ms` in the report artifact against recent nightlies | | 2 | `download` | Model could not be fetched — network, HTTP, a transfer that stalled past its 5 min budget, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | -| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner. A port conflict started no child and so has no tail: the banner carries its own remedy instead, naming the port to free — see [Running it locally](#running-it-locally) | +| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner — but only if a child actually exited. The banner says which case it is and carries the matching remedy: a port conflict started no child (see [Running it locally](#running-it-locally)), and a 5xx from a server that is still running is usually [a cut-off generation](#a-5xx-from-a-live-server), not a crash | | 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect the per-turn `used_fallback` and `replayed_opening` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | | 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | @@ -159,6 +159,32 @@ Each run writes the verdict, failure class, remedy, measured latencies and per-phase durations to the GitHub **step summary**, and uploads the full JSON report as the `model-smoke-report` artifact (30-day retention). +### A 5xx from a live server {#a-5xx-from-a-live-server} + +A 5xx is a `runtime` failure because the transport said so, but `runtime`'s +stock advice — read the child stderr tail, suspect a runner OOM or a bad +llama-cpp-python wheel — only fits a failure where a child actually died. Often +none has, and then the tail above the banner shows a perfectly healthy server. +The harness checks `poll()` on both children and, when neither has exited, prints +a different remedy: *"Neither child process exited, so nothing crashed."* + +The cause worth knowing about is a generation the adapter cut off. `convsim-core` +answers **504 `TURN_TIMEOUT`** when `CONVSIM_LLAMA_CPP_CHAT_TIMEOUT` runs out +mid-reply. The harness deliberately sets that budget *above* the latency ceiling +it is about to judge the turn on — `DEBRIEF_SLOWDOWN_FACTOR` × the ceiling, 540 s +at `--ci-hardware-factor 20` — so that a slow turn is measured and reported as +`budget` rather than killed and misreported as a crash. But that moves the cliff +rather than removing it: a turn slower than 540 s is still cut off, and with +1200 s of wall clock still unspent neither `timeout` nor `budget` takes the class +over. So exit 4 with both children alive and a 504 in the message means inference +roughly 5× slower than the nightlies have ever measured — a latency regression. +Start from `measured_ms` and `phase_durations_s` in the report artifact, not from +an OOM. + +The other way to land here with both children alive is a server that is up but +wedged: `llama-server` still loading the GGUF when readiness ran out, or a +request it refused. The stderr tails are the right place for that one. + ### Unscored debrief {#unscored-debrief} Exit 5 with *"Debrief has no rubric dimension scores"* is the one `pipeline` diff --git a/docs/real-model-smoke.md b/docs/real-model-smoke.md index d5e0e5d8..2c124f08 100644 --- a/docs/real-model-smoke.md +++ b/docs/real-model-smoke.md @@ -114,7 +114,7 @@ log, and a remedy printed next to it: | 1 | `budget` | Pipeline worked; latency regressed past the CI ceiling | Compare `measured_ms` in the report artifact against recent nightlies | | 2 | `download` | Model could not be fetched — network, HTTP, a transfer that stalled past its 5 min budget, or an empty cache | `python scripts/validate-registry.py --url-check`, then re-run | | 3 | `checksum` | **SHA-256 drift**: on-disk bytes ≠ `model-registry/registry.yaml` | See [Checksum drift](#checksum-drift) — never relax the check | -| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner. A port conflict started no child and so has no tail: the banner carries its own remedy instead, naming the port to free — see [Running it locally](#running-it-locally) | +| 4 | `runtime` | `llama-server` or `convsim-core` crashed, hung, or returned 5xx — or one of the two ports was already taken, so neither could be started | Read the child stderr tail printed above the banner — but only if a child actually exited. The banner says which case it is and carries the matching remedy: a port conflict started no child (see [Running it locally](#running-it-locally)), and a 5xx from a server that is still running is usually [a cut-off generation](#a-5xx-from-a-live-server), not a crash | | 5 | `pipeline` | Servers healthy, but an end-to-end assertion failed | Inspect the per-turn `used_fallback` and `replayed_opening` flags in the report artifact — except for an unscored debrief, see [Unscored debrief](#unscored-debrief) | | 6 | `timeout` | Wall-clock budget exhausted; the failing phase is named | Check `phase_durations_s` before raising the budget | @@ -155,6 +155,32 @@ Each run writes the verdict, failure class, remedy, measured latencies and per-phase durations to the GitHub **step summary**, and uploads the full JSON report as the `model-smoke-report` artifact (30-day retention). +### A 5xx from a live server + +A 5xx is a `runtime` failure because the transport said so, but `runtime`'s +stock advice — read the child stderr tail, suspect a runner OOM or a bad +llama-cpp-python wheel — only fits a failure where a child actually died. Often +none has, and then the tail above the banner shows a perfectly healthy server. +The harness checks `poll()` on both children and, when neither has exited, prints +a different remedy: *"Neither child process exited, so nothing crashed."* + +The cause worth knowing about is a generation the adapter cut off. `convsim-core` +answers **504 `TURN_TIMEOUT`** when `CONVSIM_LLAMA_CPP_CHAT_TIMEOUT` runs out +mid-reply. The harness deliberately sets that budget *above* the latency ceiling +it is about to judge the turn on — `DEBRIEF_SLOWDOWN_FACTOR` × the ceiling, 540 s +at `--ci-hardware-factor 20` — so that a slow turn is measured and reported as +`budget` rather than killed and misreported as a crash. But that moves the cliff +rather than removing it: a turn slower than 540 s is still cut off, and with +1200 s of wall clock still unspent neither `timeout` nor `budget` takes the class +over. So exit 4 with both children alive and a 504 in the message means inference +roughly 5× slower than the nightlies have ever measured — a latency regression. +Start from `measured_ms` and `phase_durations_s` in the report artifact, not from +an OOM. + +The other way to land here with both children alive is a server that is up but +wedged: `llama-server` still loading the GGUF when readiness ran out, or a +request it refused. The stderr tails are the right place for that one. + ### Unscored debrief Exit 5 with *"Debrief has no rubric dimension scores"* is the one `pipeline` diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 86a9ea5c..735dffb3 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1464,6 +1464,44 @@ def timeout_latency_evidence( ) +def runtime_without_crash_remedy(llama_timeout_s: float) -> str: + """The ``runtime`` advice for a failure where neither child actually died. + + ``runtime``'s class remedy opens with "The child stderr tail is printed above + the banner" and lists crash causes — runner OOM on model load, an + incompatible wheel, an unhandled exception. All of that assumes a child + exited. Often none has: every transport-level failure is a ``runtime`` + failure, so a *live* convsim-core that answered 5xx, or a live llama-server + still loading when readiness ran out, lands in this class with both children + running and a stderr tail that shows a healthy server. + + The 5xx is the one worth naming, because the cause behind it is usually not + a crash at all. ``_start_core`` raises the adapter's generation budget + above the latency ceiling precisely so a slow turn is *measured* rather than + killed — but that moves the cliff, it does not remove it. A turn slower + than that budget is still cut off adapter-side and convsim-core answers + 504 ``TURN_TIMEOUT``, and since the budget is well inside the wall clock + (540 s against 1200 s at ``--ci-hardware-factor 20``) the run still has time + left, so neither ``timeout`` nor ``budget`` takes the class over. The + verdict is then honest about the transport and wrong about the cause, with a + remedy sending triage after an OOM that did not happen — which is the same + misclassification ``_start_core``'s raised timeout exists to prevent, one + cliff further out. + """ + return ( + "Neither child process exited, so nothing crashed: the stderr tails above are " + "from servers that were still running when the run gave up. A live server " + "either never became ready or did not answer one request. If convsim-core " + "answered 504 TURN_TIMEOUT, that is the adapter's own generation budget " + f"({llama_timeout_s:.0f} s, CONVSIM_LLAMA_CPP_CHAT_TIMEOUT, which this harness " + "sets above the latency ceiling for exactly this reason) running out: " + "inference slower than that is a latency regression, not a crash, so compare " + "measured_ms and phase_durations_s in the report artifact against recent " + "nightlies instead of looking for an OOM. Otherwise read the tails for a " + "server that is alive but wedged — a model still loading, a request refused." + ) + + # ── Reporting ───────────────────────────────────────────────────────────────── @@ -2024,6 +2062,19 @@ def run_smoke( results["remedy"] = ( exc.remedy if failure_class == exc.failure_class else REMEDIES[failure_class] ) + # A `runtime` verdict with both children still alive did not come from a + # crash, and the class remedy's first instruction — read the child stderr + # tail — is advice about a tail that shows a healthy server. The common + # cause is a generation the adapter cut off, i.e. a latency regression. + # See runtime_without_crash_remedy. Only when the failure was using the + # class default: the port-conflict failure already carries a remedy of + # its own, and it is right about a run where no child was ever started. + if ( + failure_class == FailureClass.RUNTIME + and not crashed + and results["remedy"] == REMEDIES[FailureClass.RUNTIME] + ): + results["remedy"] = runtime_without_crash_remedy(llama_timeout_s) if not results["failures"]: results["failures"] = [str(exc)] if crashed: diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 6db98190..8eda8ad0 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -1354,6 +1354,39 @@ def test_the_restored_and_saved_cache_entries_are_the_same(self) -> None: ) +class TestLiveServerRemedyIsDocumented: + """The runtime row of the doc's failure table sends the reader somewhere real. + + The remedy printed for a `runtime` failure with both children alive is three + sentences in a banner; the reasoning behind it — why a 504 is a latency + regression rather than a crash, and what the adapter budget has to do with it + — lives in the doc. The row links to that section, so the section has to + exist in both copies. + """ + + @pytest.mark.parametrize("doc", _REAL_MODEL_SMOKE_DOCS, ids=lambda p: p.parts[0]) + def test_the_failure_table_links_a_section_that_exists(self, doc: Path) -> None: + text = doc.read_text(encoding="utf-8") + assert "(#a-5xx-from-a-live-server)" in text, ( + "the runtime row no longer links the live-server section" + ) + assert "### A 5xx from a live server" in text + + @pytest.mark.parametrize("doc", _REAL_MODEL_SMOKE_DOCS, ids=lambda p: p.parts[0]) + def test_the_documented_adapter_budget_is_the_one_the_harness_sets( + self, doc: Path + ) -> None: + # Both the doc and the remedy quote a number the harness computes, so + # either can drift from it silently. 540 s is the budget at the factor + # the workflow passes. + ceiling_s = ( + smoke.BUDGETS_MS["full_response_ms"] * 20 * smoke.REGRESSION_TOLERANCE + ) / 1000 + expected = max(300.0, ceiling_s * smoke.DEBRIEF_SLOWDOWN_FACTOR + 60.0) + assert f"{expected:.0f} s" in doc.read_text(encoding="utf-8") + assert f"{expected:.0f} s" in smoke.runtime_without_crash_remedy(expected) + + class TestWheelInstallInvariants: """llama-cpp-python must arrive as a prebuilt wheel, never as a source build. @@ -2514,6 +2547,87 @@ def _boom(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: results = json.loads(report.read_text(encoding="utf-8")) assert results["failure_class"] == smoke.FailureClass.RUNTIME assert "convsim-core exited with code 1" in results["failures"][0] + # A child really did die, so the class remedy's "read the stderr tail" + # is the right advice and must not be swapped for the live-server one. + assert results["remedy"] == smoke.REMEDIES[smoke.FailureClass.RUNTIME] + + def test_a_live_server_answering_5xx_is_not_blamed_on_a_crash( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + """A `runtime` verdict with both children alive must not send triage after an OOM. + + Every transport-level failure is a `runtime` failure, so a *live* + convsim-core that answered 504 TURN_TIMEOUT lands in the same class as a + crash — and the class remedy opens by telling the reader to read a child + stderr tail that shows a healthy server, then lists runner OOM and a bad + wheel as the likely causes. + + The realistic cause is neither. `_start_core` raises the adapter's + generation budget above the latency ceiling so a slow turn is measured + rather than killed, but that only moves the cliff: a turn slower than + 540 s (the budget at --ci-hardware-factor 20) is still cut off + adapter-side, and with 1200 s of wall clock the run still has time left, + so neither `timeout` nor `budget` takes the class over. + """ + models_dir, model_id, digest = staged_model + + def _timed_out(url: str, *, payload=None, timeout=None, expect=(200, 201)) -> dict: + if url.endswith("/health"): + return {"llm_runtime": {"runtime_id": "llama_cpp"}} + if url.endswith("/api/sessions"): + return {"session_id": "sess-1"} + if url.endswith("/start"): + return _OPENING + raise smoke.SmokeFailure( + smoke.FailureClass.RUNTIME, + f"{url} returned HTTP 504: TURN_TIMEOUT", + ) + + monkeypatch.setattr(smoke, "_request_json", _timed_out) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.RUNTIME] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["failure_class"] == smoke.FailureClass.RUNTIME + remedy = results["remedy"] + assert remedy != smoke.REMEDIES[smoke.FailureClass.RUNTIME] + assert "Neither child process exited" in remedy + assert "CONVSIM_LLAMA_CPP_CHAT_TIMEOUT" in remedy + # The number in the advice has to be the budget the harness actually set + # on the adapter, or the reader cannot tell a cut-off turn from a crash. + assert "540 s" in remedy + # And it has to send the reader at the latency measurements, which is + # where the cause actually shows up. + assert "measured_ms" in remedy and "phase_durations_s" in remedy + + def test_a_port_conflict_keeps_its_own_remedy( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # The live-server remedy above must not displace this one: a port + # conflict starts no child at all, so "neither child process exited" is + # true but useless, and the advice the reader needs is the port to free. + models_dir, model_id, digest = staged_model + monkeypatch.setattr( + smoke, "_assert_port_free", + lambda port, label: (_ for _ in ()).throw(smoke.SmokeFailure( + smoke.FailureClass.RUNTIME, + f"Port {port} is already in use, so {label} cannot be started on it", + remedy="Stop whatever is listening on it, then re-run.", + )), + ) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == smoke.EXIT_CODES[smoke.FailureClass.RUNTIME] + results = json.loads(report.read_text(encoding="utf-8")) + assert results["remedy"] == "Stop whatever is listening on it, then re-run." def test_a_fake_runtime_is_refused( self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path From 0ec77e6d33a2df2f8241fa78f5a85e3e95a7fc37 Mon Sep 17 00:00:00 2001 From: Nick Charney Kaye Date: Sat, 3 Oct 2026 11:54:02 -0400 Subject: [PATCH 58/58] Stop a flagless debug entry from reading as a passed fallback check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `_debug_flags_by_turn` drops any /debug entry it cannot key by turn number, and `evaluate_turns` warns per turn that the fallback check did not run for the turns left without flags — the whole point being that an absent `used_fallback` is falsy, so an unflagged turn would otherwise count towards "at least one generated turn was a real reply" on no evidence. An entry that *is* keyed but carries no `used_fallback` slipped straight through that: the copy used `flag.get(...)`, which wrote False, and `evaluate_turns` decides whether the check ran from the key's presence, so the turn was both credited as a real reply and stripped of the warning that says nothing checked it. Verified against the faked servers: a payload of bare `{"turn_number": n}` entries produced `used_fallback: false` on all three turns, zero warnings, exit 0 — a run reporting a proof it never obtained. `used_native_structured_output` recorded the same invented False into the report artifact. Copy each flag only when the payload carries it, so a turn the endpoint did not describe stays unflagged and the existing warning fires. Two tests cover it: all flags missing, and an entry carrying `used_fallback` alone, which must still prove the turn. Co-Authored-By: Claude Opus 5 --- scripts/nightly-model-smoke.py | 19 +++++-- tests/scripts/test_nightly_model_smoke.py | 61 +++++++++++++++++++++++ 2 files changed, 75 insertions(+), 5 deletions(-) diff --git a/scripts/nightly-model-smoke.py b/scripts/nightly-model-smoke.py index 735dffb3..1bb67b4b 100755 --- a/scripts/nightly-model-smoke.py +++ b/scripts/nightly-model-smoke.py @@ -1939,11 +1939,20 @@ def run_smoke( flags = _debug_flags_by_turn(base, session_id, clock.cap(30.0)) for turn in results["turns"]: flag = flags.get(turn["turn_number"]) - if flag: - turn["used_fallback"] = bool(flag.get("used_fallback")) - turn["used_native_structured_output"] = bool( - flag.get("used_native_structured_output") - ) + if not flag: + continue + # Only the flags the payload actually carries. `.get()` would read + # an absent flag as False, which is not "the model did not fall + # back" — it is "we were not told" — and writing the key at all is + # worse than recording the wrong value: evaluate_turns decides + # whether the fallback check ran by whether the key is *present*, + # so a payload that keys a turn but omits the flag would both + # credit the turn as a real reply and suppress the warning that + # says nothing checked. Same hole _debug_flags_by_turn closes for + # an entry it cannot key, one level down. + for name in ("used_fallback", "used_native_structured_output"): + if name in flag: + turn[name] = bool(flag[name]) # ── debrief ─────────────────────────────────────────────────────────── clock.enter("debrief") diff --git a/tests/scripts/test_nightly_model_smoke.py b/tests/scripts/test_nightly_model_smoke.py index 8eda8ad0..19bb7bab 100644 --- a/tests/scripts/test_nightly_model_smoke.py +++ b/tests/scripts/test_nightly_model_smoke.py @@ -3021,6 +3021,67 @@ def test_a_debug_payload_that_keys_only_some_turns_says_which_it_skipped( assert "1 of 3" in warning assert "player_turn_1" in warning + def test_a_debug_entry_without_the_flags_is_not_read_as_a_passed_check( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # The other way a /debug payload can leave the fallback check unrun: an + # entry that *is* keyed by turn number but carries no used_fallback. + # Reading that as False would credit every turn as a real reply on no + # evidence — and because evaluate_turns decides whether the check ran + # from the key's presence, writing it would also suppress the warning + # that says so, which is strictly worse than the unkeyable-entry case + # above. The turn must come back unflagged, and the run must say it is + # unproven. + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core( + _debrief(), + debug_turns=[{"turn_number": n} for n in range( + 1, len(smoke.SCRIPTED_PLAYER_TURNS) + 1 + )], + )) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == 0 + results = json.loads(report.read_text(encoding="utf-8")) + generated = [t for t in results["turns"] if t["model_generated"]] + assert generated + for turn in generated: + assert "used_fallback" not in turn, turn + assert "used_native_structured_output" not in turn, turn + assert any("fallback check did not run" in w for w in results["warnings"]) + + def test_a_debug_entry_with_only_some_flags_keeps_the_one_it_carries( + self, staged_model, fake_servers, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # Per flag, not per entry: used_fallback is the one the verdict turns + # on, so an entry carrying it must still prove the turn even when the + # structured-output flag beside it is gone. + models_dir, model_id, digest = staged_model + monkeypatch.setattr(smoke, "_request_json", _fake_core( + _debrief(), + debug_turns=[ + {"turn_number": n, "used_fallback": False} + for n in range(1, len(smoke.SCRIPTED_PLAYER_TURNS) + 1) + ], + )) + report = tmp_path / "report.json" + + exit_code = smoke.run_smoke( + model_id, 20.0, report, model_sha256=digest, models_dir=models_dir + ) + + assert exit_code == 0 + results = json.loads(report.read_text(encoding="utf-8")) + generated = [t for t in results["turns"] if t["model_generated"]] + for turn in generated: + assert turn["used_fallback"] is False + assert "used_native_structured_output" not in turn, turn + assert not any("fallback check did not run" in w for w in results["warnings"]) + # --------------------------------------------------------------------------- # CLI entry point