Play a real-model scripted conversation nightly and assert a scored debrief

.github/workflows/ci.yml

@@ -237,6 +237,15 @@ jobs:
         run: |
           echo "Local: python -m pytest tests/acceptance/test_network_guard.py -v"
           python -m pytest tests/acceptance/test_network_guard.py -v
+      # The real-model nightly (model-smoke-nightly.yml) cannot run per-PR, but
+      # its decision logic can: checksum verification, the end-to-end
+      # assertions, budget arithmetic and failure classification.  Without this
+      # the harness is only ever exercised at 04:00 UTC, so a typo in it shows up
+      # as a mysteriously red nightly.
+      - name: Run nightly real-model smoke harness unit tests (no model needed)
+        run: |
+          echo "Local: python -m pytest tests/scripts/ -v"
+          python -m pytest tests/scripts/ -v
 
   desktop-check:
     name: Desktop Rust check

.github/workflows/model-smoke-nightly.yml

@@ -1,33 +1,100 @@
 # SPDX-License-Identifier: Apache-2.0
-# Nightly real-model smoke test.
+# Nightly real-model smoke test (issue #457).
 #
-# Downloads the pinned starter model from the registry, starts a local
-# llama-server (llama-cpp-python's OpenAI-compatible server) plus convsim-core
-# wired to it, runs a scripted end-to-end session through convsim-core's REST
-# API, measures full-response latency, and fails if it regresses more than
-# 20 % above the documented budget.
+# Nightly, NOT per-PR: this is the only CI job that plays a conversation on a
+# real local model.  Every other job validates against the fake/scripted
+# runtimes, so a regression that only shows up under real inference — a prompt
+# that the model cannot satisfy, a grammar the adapter mis-serialises, a
+# debrief that never scores — is invisible until this job runs.
 #
-# convsim-core's turn endpoint is synchronous (token streaming lives in the
-# apps/api gateway, which is not stood up here), so the smoke measures full
-# end-to-end turn latency against the FULL_RESPONSE_MS budget (10 s).
+# What it does: download the smallest registry model (role `starter`, Qwen3 4B
+# Q4_K_M, ~2.5 GB) with SHA-256 verification, start llama-cpp-python's
+# OpenAI-compatible server on it, start convsim-core wired to that server
+# (CONVSIM_RUNTIME_ID=llama_cpp), play a scripted multi-turn conversation
+# through convsim-core's REST API, and assert a SCORED debrief comes back.
+# Latency is checked against the documented budgets.  Full detail, including
+# the local repro command: docs/real-model-smoke.md.
 #
-# The model file (~2.5 GB) is cached by its registry SHA-256 so repeated
-# nightly runs skip the download.  The first run after a pinned-URL change
-# re-downloads and re-caches the new file.
+# ── Failure classification ───────────────────────────────────────────────────
+# scripts/nightly-model-smoke.py attributes every failure to one class and
+# exits with a distinct code, so a red nightly is triageable from the job
+# summary without reading logs:
 #
-# Timings are measured on GitHub-hosted ubuntu-latest (2 vCPU, ~8 GB RAM,
-# CPU-only inference).  The CI budget is the documented budget multiplied by
-# a hardware-tier factor that accounts for the slower runner:
+#   1  budget    — pipeline worked, latency regressed
+#   2  download  — model could not be fetched (network/HTTP/stall/empty cache)
+#   3  checksum  — SHA-256 drift against the registry
+#   4  runtime   — llama-server or convsim-core crashed, hung, or 5xx'd
+#   5  pipeline  — servers healthy, end-to-end assertion failed
+#   6  timeout   — wall-clock budget exhausted (failing phase named)
+#
+# The script writes the class, the failures, the remedy, per-phase durations
+# and measured latencies to $GITHUB_STEP_SUMMARY, and the full JSON report is
+# uploaded as an artifact.
+#
+# ── Runtime budget: target < 30 min on standard GitHub runners ───────────────
+# timeout-minutes: 30 is the hard ceiling; the script's own
+# --wall-clock-budget-s 1200 (20 min) trips first so the failure is attributed
+# to a phase instead of surfacing as an unattributable job cancellation.  The
+# 10 min gap is deliberate margin, not an accounting of known work: the steps
+# before the smoke (checkout, pip install, cache restore and, on a cache miss,
+# the download plus the cache save) run against the same job clock and are
+# invisible to the script's own deadline, but they measure 31–51 s in total.
+# The margin is sized for the network-bound part of that, which a slow
+# Hugging Face or Actions-cache night can stretch by minutes.  The download is
+# the one of those steps that could stretch without limit, so it carries its own
+# 5 min budget (DOWNLOAD_BUDGET_S in the script) and fails as `download` rather
+# than spending the margin the smoke still needs.
+#
+#   set up job + checkout + setup-python     ~5 s
+#   pip install ×3 (cached wheels)           ~16 s cold, ~14 s warm
+#   model cache restore (2.3 GiB)            ~24 s on a hit, <1 s on a miss
+#   download + SHA-256 verify (cache miss)   ~10 s    — bounded at 5 min
+#   cache save (cache miss only)             ~10 s
+#   SHA-256 re-verify before load            <10 s    — bounded, see below
+#   model load + core start + session        ~8 s
+#   3 scripted turns                         ~6 min   — projected, see below
+#   debrief generation                       ~4 min   — projected, see below
+#   ------------------------------------------------
+#   total                                   ~11 min, warm cache or cold
+#
+# The non-inference figures are measured, from this job's own logs for the
+# twelve nightlies whose logs GitHub still held on 2026-10-03.  Two of them are
+# counter-intuitive and worth knowing before tuning anything: a cache MISS is no
+# more expensive than a cache HIT — ~10 s to fetch and hash the GGUF plus ~10 s
+# to save it, against ~19–28 s to restore the same bytes — so the cache saves
+# the network and not the clock; and everything before the conversation is
+# 31–51 s of a 30 min job, so the budget is inference and nothing else.
+# The re-verify row is the one with no direct measurement — the previous
+# single-turn harness did not re-verify — and is bounded by the cold row above,
+# where downloading 2.3 GiB *and* hashing it together took 10 s.
+#
+# The inference figures are projected from the one latency this job has
+# actually measured — a single behavioral-interview turn, twelve nightlies:
+# 66/71/93/94/98/103/104/111/112/114/115/115 s — rounded up to ~120 s per turn
+# because turns 2 and 3 render a longer transcript than turn 1, with the
+# debrief allowed twice a turn.  Replace both with real per-phase durations
+# from the first green nightly's report artifact.
+#
+# If the smoke step creeps past ~15 min — or the job past ~25 min — reduce
+# SCRIPTED_PLAYER_TURNS rather than raising either ceiling: a nightly that
+# routinely runs near its timeout flaps.
+#
+# ── Latency budget on CI hardware ────────────────────────────────────────────
+# Timings are measured on GitHub-hosted ubuntu-latest, CPU-only inference.
+# The CI budget is the documented budget multiplied by a hardware-tier factor:
 #
 #   CI budget = documented budget × CI_HARDWARE_FACTOR (currently 20)
 #
 # The factor is calibrated empirically: a full behavioral-interview turn on
-# this runner measures ~116 s (two consecutive runs at 117.0 s and 115.3 s —
-# large grammar-constrained prompt, 4B-Q4 model, CPU-only), i.e. ~11.6× the
-# 10 s documented budget.  20× gives a 10 s × 20 = 200 s CI budget and, with
-# the script's 1.20 regression tolerance, a 240 s ceiling — roughly 2× headroom
-# over the observed latency so normal runner-to-runner variance does not flap
-# the nightly, while a genuine >2× regression still fails.  This factor only
+# this runner has measured 66–115 s across the twelve nightlies whose logs
+# GitHub still held on 2026-10-03 (median ~104 s — large grammar-constrained
+# prompt, 4B-Q4 model, CPU-only), i.e. ~11.5× the 10 s documented budget at the
+# slow end, which is the end the factor is set from.  20× gives a
+# 10 s × 20 = 200 s CI budget and, with the script's 1.20 regression tolerance,
+# a 240 s ceiling — roughly 2× headroom over the observed latency so normal
+# runner-to-runner variance does not flap the nightly, while a genuine >2×
+# regression still fails.  The headline full_response_ms is the MEDIAN of the
+# scripted turns, so one unlucky turn cannot flap the job.  This factor only
 # models CI-hardware slowness; the product's 10 s target-hardware SLO is
 # unchanged.  Re-measure and re-tune if the runner class or starter model
 # changes.
@@ -42,11 +109,19 @@ on:
 permissions:
   contents: read
 
+env:
+  # Bump this prefix to abandon a poisoned model cache entry (Actions cache
+  # entries are immutable, so a corrupt GGUF under a given key would otherwise
+  # be restored on every re-run — see the checksum remedy in the script).
+  MODEL_CACHE_PREFIX: model-gguf-v1
+
 jobs:
   model-smoke:
     name: Starter-model end-to-end smoke
     runs-on: ubuntu-latest
-    timeout-minutes: 60
+    # Hard ceiling for the documented < 30 min runtime budget.  The script's
+    # own 20 min wall-clock budget trips first and names the failing phase.
+    timeout-minutes: 30
 
     steps:
       - uses: actions/checkout@v5
@@ -65,42 +140,44 @@ jobs:
       - name: Install convsim-core
         run: pip install -e "services/convsim-core[dev]"
 
+      # --only-binary enforces what the step name already promises.  PyPI ships
+      # llama-cpp-python as an sdist only — the prebuilt wheels live on the
+      # extra index — and --extra-index-url does not prefer them: pip resolves
+      # the newest version across both indexes, so a release the wheel index has
+      # not caught up with yet is installed by compiling llama.cpp on the
+      # runner.  That build turns the ~16 s install row in the runtime budget
+      # into 15+ min, which the 30 min job timeout has no room for next to ~11
+      # min of inference: the job would die
+      # with the unattributable cancellation this whole design exists to avoid,
+      # on a night when nothing about the product changed.  Refusing the sdist
+      # makes pip fall back to the newest version that does have a wheel, and if
+      # there is none it fails in seconds against this step's own name.
       - name: Install llama-cpp-python with server extra (CPU-only prebuilt wheel)
         run: |
           pip install "llama-cpp-python[server]" \
+            --only-binary llama-cpp-python \
             --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
 
-      - name: Read starter model metadata from registry
+      # Single source of truth: the cache key, the download URL and the
+      # checksum the smoke verifies against all come from this one lookup, so
+      # they cannot drift apart.
+      - name: Resolve starter model from the registry
         id: registry
         run: |
-          python - <<'EOF'
-          import yaml, json, sys
-          from pathlib import Path
-          reg = yaml.safe_load(Path("model-registry/registry.yaml").read_text())
-          starter = next(m for m in reg["models"] if m["role"] == "starter")
-          url = starter["download"]["url"]
-          sha256 = starter["download"]["sha256"]
-          model_id = starter["id"]
-          print(f"model_id={model_id}")
-          print(f"model_url={url}")
-          print(f"model_sha256={sha256}")
-          # Write to GITHUB_OUTPUT
-          import os
-          out = os.environ.get("GITHUB_OUTPUT", "/dev/stdout")
-          with open(out, "a") as f:
-              f.write(f"model_id={model_id}\n")
-              f.write(f"model_url={url}\n")
-              f.write(f"model_sha256={sha256}\n")
-          EOF
-
-      - name: Cache model file
+          echo "Local: python scripts/nightly-model-smoke.py --print-registry-model starter"
+          python scripts/nightly-model-smoke.py --print-registry-model starter
+
+      # Restore/save are split (rather than actions/cache) so a model that
+      # fails checksum verification is never written to the cache: the save
+      # step is gated on success().
+      - name: Restore cached model file
         id: model-cache
-        uses: actions/cache@v5
+        uses: actions/cache/restore@v5
         with:
           path: ~/.convsim/models/llm
-          key: model-${{ steps.registry.outputs.model_sha256 }}
+          key: ${{ env.MODEL_CACHE_PREFIX }}-${{ steps.registry.outputs.model_sha256 }}
 
-      - name: Download starter model
+      - name: Download starter model (SHA-256 verified)
         if: steps.model-cache.outputs.cache-hit != 'true'
         run: |
           python scripts/nightly-model-smoke.py \
@@ -109,11 +186,24 @@ jobs:
             --model-sha256 "${{ steps.registry.outputs.model_sha256 }}" \
             --model-id "${{ steps.registry.outputs.model_id }}"
 
-      - name: Run end-to-end smoke and check latency budgets
+      - name: Cache the verified model file
+        if: success() && steps.model-cache.outputs.cache-hit != 'true'
+        uses: actions/cache/save@v5
+        with:
+          path: ~/.convsim/models/llm
+          key: ${{ env.MODEL_CACHE_PREFIX }}-${{ steps.registry.outputs.model_sha256 }}
+
+      # --model-sha256 makes the run re-verify the file it is about to load, so
+      # a drifted or truncated CACHE HIT fails loudly (exit 3) instead of
+      # feeding corrupt weights to the smoke.
+      - name: Run real-model smoke (scripted conversation + scored debrief)
         run: |
+          echo "Local: see docs/real-model-smoke.md for the one-line local repro"
           python scripts/nightly-model-smoke.py \
             --model-id "${{ steps.registry.outputs.model_id }}" \
+            --model-sha256 "${{ steps.registry.outputs.model_sha256 }}" \
             --ci-hardware-factor 20 \
+            --wall-clock-budget-s 1200 \
             --report-path /tmp/smoke-report.json
 
       - name: Upload smoke report

ROADMAP.md

@@ -91,7 +91,7 @@ criteria.
 
 - [x] Real UI screenshots and demo recording, captured from a local-model playthrough — [#455](https://github.com/outrightmental/ConversationSimulator/issues/455)
 - [ ] Desktop app with bundled backend (Tauri sidecar for `convsim-core`) — [#456](https://github.com/outrightmental/ConversationSimulator/issues/456)
-- [ ] Automated real-model CI smoke test — [#457](https://github.com/outrightmental/ConversationSimulator/issues/457)
+- [x] Automated real-model CI smoke test — [#457](https://github.com/outrightmental/ConversationSimulator/issues/457)
 
 ### Post-alpha — Milestone 2+
 

docs-site/src/content/docs/dev/adr-0001-npc-relationship-memory.md

@@ -2,7 +2,7 @@
 title: "ADR-0001: NPC relationship memory"
 description: "Spike go/no-go report on persistent NPC relationship memory, covering the prototype, evaluation results, safety and privacy reviews, and a conditional-go recommendation."
 sidebar:
-  order: 14
+  order: 15
 ---
 
 **Status:** Proposed (spike branch — not merged)  

docs-site/src/content/docs/dev/offline-smoke-tests.md

@@ -131,6 +131,10 @@ Exit codes: `0` = pass, `1` = network violation or pack error, `3` = unexpected
 To verify that the installed llama.cpp or Ollama runtime makes no *unexpected*
 outbound calls, run the smoke test while the real runtime is active.
 
+> This page covers the **network** promise only. For the nightly CI job that
+> plays a scripted conversation on a real model end-to-end and asserts a scored
+> debrief, see [Real-model smoke test](/dev/real-model-smoke/).
+
 ### 1. Start the local runtime
 
 ```bash

docs-site/src/content/docs/dev/post-alpha-issues.md

@@ -124,7 +124,19 @@ latency and output quality signals.
 cache-unfriendly in most CI environments. The fake runtime provides full
 structural coverage; real-model CI is a quality-of-life improvement.
 
-**Milestone:** 2 (CI hardening)
+**Milestone:** 2 (CI hardening)  
+**Status:** ✅ **Shipped,** pulled forward into the Milestone-1 polish set
+rather than waiting for Milestone 2.
+[`.github/workflows/model-smoke-nightly.yml`](https://github.com/outrightmental/ConversationSimulator/blob/main/.github/workflows/model-smoke-nightly.yml)
+runs nightly: it resolves the registry's `starter` model from one lookup,
+downloads it with SHA-256 verification (cached between runs and re-verified
+before the weights load, so a drifted cache hit fails loudly), starts
+`llama-server` and `convsim-core` on it, plays a scripted multi-turn
+conversation and asserts a *scored* debrief — inside a documented budget under
+30 min. Every failure is attributed to one of six classes with its own exit
+code, so a red nightly is triageable from the job summary; the harness's own
+decision logic is unit-tested per-PR. See
+[Real-model smoke test](/dev/real-model-smoke/).
 
 ---