[VerifiedOrganic]

Summary

P4 of the 0.4 semantic continuity lane: evaluator integration for --semantic off|shadow|on, live five-metric evaluation over the four public sets, the no-route host round-trip acceptance path, and documentation/changelog updates. The default remains semantic_mode = off.

What changed

• evals/continuity_recall.py accepts --semantic and --semantic-judge-identity for run, held-out, held-out-2, and held-out-3.
• New evals/semantic_eval.py seeds each set once, runs shadow first, then on against the same projects (cache-backed), and reports:
  • deterministic-only vs semantic-on localized and type-present recall;
  • semantic warning precision, control warnings by check type, evidence-anchor validity;
  • repair rate, calls per candidate scene, cache-hit rate;
  • p95 prompt/output tokens and latency, token totals per scene/chapter;
  • would-be warning/info counts for shadow/on parity.
• Added per-scene semantic accounting to check_consistency output (semantic_scene_summaries), with bounded/schema/profile updates.
• Checked-in reports: evals/results/semantic_p4_*.json.
• The no-route host round-trip acceptance is covered by the P3 host_report_emits_in_on_mode_and_shadow_counts_without_findings test: prepare request → host report → validated findings, with no judge route configured.
• Docs: semantic design results section and tool-profile usage; CHANGELOG under Unreleased (0.4.0). No migration.

Live evaluation

Judge config held constant throughout: ollama:gemma4:31b via ai run OLLAMA_API_KEY -- .... The evaluator runs shadow first, then on against the same projects; the on pass reuses the validated cache. P95s are per actual dispatch receipt, including failed/timeout dispatches; scene totals are retained for cost.

Set	Deterministic recall	Semantic-on recall	Semantic warnings	Control warnings	Cache hit (on)	Repair rate (shadow/on)	p95 prompt/call	p95 output/call	p95 latency/call	Tokens/scene (on)
Baseline	28/28	28/28	0	unchanged (2 warnings + 1 info)	78.8%	19.2% / 48.5%	5,148	1,200	5,870 ms	1,887
#72	54/56	54/56	0	unchanged (0)	48.4%	34.0% / 50.0%	6,609	1,200	6,637 ms	6,862
held-out-2	55/56 localized, 56/56 type	55/56, 56/56	0	unchanged (0)	59.3%	29.8% / 48.9%	4,775	1,200	3,060 ms	3,500
held-out-3	26/56 localized, 28/56 type	26/56, 28/56	0	unchanged (0)	34.4%	39.6% / 50.0%	6,878	1,200	4,111 ms	9,526

Why zero warnings, including info?

The judge returned zero contradiction verdicts on every set. All validated judgments were compatible or insufficient, and every confidence landed in the top bucket (>=0.85):

• Baseline: 257 compatible / 92 insufficient (shadow); 271 / 93 (on)
• feat(evals): held-out continuity contradiction corpus #72: 162 / 38; held-out-2: 112 / 48; held-out-3: 100 / 19
• Only discard reason: not_contradiction
• No evidence-anchor, canon-ref, confidence-threshold, or dedupe discards
• Stop reasons were success/truncated (truncated: baseline 7/99 calls, feat(evals): held-out continuity contradiction corpus #72 52/97, held-out-2 10/84, held-out-3 75/106 shadow); successful calls also contained no contradiction

Nothing was discarded downstream: there was no contradiction to promote, downgrade, or emit as info. Truncation/repair rates are a quality problem but are not the cause of the zero-warning result.

Activation gate fails: no semantic recall lift, warning precision is null (zero denominator), evidence validity is vacuous, and repair rates (19-50%) exceed <=10%. Per-call prompt/output/latency ceilings pass; control warnings did not rise. Default remains off.

Cost: no provider price is configured, so no dollar figure was fabricated. Held-out-4 was not accessed.

Gates

• Full workspace tests, clippy -D warnings, and fmt: pass (p4_final2 run; clippy allow for the per-call helper only).
• Evaluator precision fixtures (semantic_eval_self_test), CLI self-test, and P3 host/cache tests: pass.
• CI pending on this PR.
• Default remains off; checkpoint posted; wait for maintainer merge.

[VerifiedOrganic]

Changes requested. The evaluator currently computes semantic warning precision from plant recall and compares per-scene totals with per-call activation limits. Those measurement errors need correction before the checked-in gate interpretation is reliable. Required changes are inline. Reviewed 7840dee against base 54503f4. Static inspection only; no tests, builds, or project programs executed.

[VerifiedOrganic]

Changes requested. Warning-instance precision/null handling and successful-call token percentiles are fixed. The remaining per-call measurement finding is not fully resolved: repair rate uses candidate scenes as its denominator, and failed/time-out dispatches do not receive receipts and vanish from metrics. Required follow-ups are inline. Reviewed 0226931 against base 54503f4, including the latest diff. Static inspection only; tests and project programs were not executed.

[VerifiedOrganic]

Approved. The remaining accounting findings are addressed: repair_rate is repair dispatches / all dispatches, repaired-scene rate is reported separately, and failed/time-out dispatches now contribute receipts with elapsed time and reserved input usage. Reports/docs were regenerated, and an offline metric fixture distinguishes call-level statistics from scene totals. The prior warning-instance precision and per-call percentile fixes remain intact. Reviewed df38fb7 against base 54503f4, including the latest follow-up diff. No further required changes found. Static inspection only; tests and project programs were not executed.