Skip to content

feat(semantic): add P3 judge transport and cache - #100

Merged
VerifiedOrganic merged 7 commits into
mainfrom
ao/planning-agent-entry/semantic-p3
Nov 11, 2026
Merged

VerifiedOrganic merged 7 commits into
mainfrom
ao/planning-agent-entry/semantic-p3

Conversation

@VerifiedOrganic

@VerifiedOrganic VerifiedOrganic commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

P3 of the 0.4 semantic continuity lane: the continuity_semantic judge route with review fallback, strict prompt/parser with one repair, typed fail-closed errors, validated-report caching, host-report/report-side validation, and shadow/on accounting. Default remains semantic_mode = off.

What changed

  • Routing: added continuity_semantic to the prose-bearing routes. It resolves before review; a missing/rating-denied route falls back to review, and a built-in local stub is always rejected as a judge. Typed RouteUnavailable / RatingDenied errors are recorded as not-evaluated coverage.
  • Prompt/parser: one bounded prompt per scene containing only the scene identity, ranked candidates, canon refs, excerpts, hints, and the strict SemanticJudgeReportV1 output contract. The parser accepts one fenced JSON object, rejects unknown fields, validates against the core contracts, and performs at most one repair with compact validation errors. Transport/timeout errors do not consume the repair.
  • Cache: migration V0058 adds semantic_continuity_cache, keyed by scene hash, canon version, candidate-set hash, contract version, route, and model identity. Only validated typed reports are stored; corrupt entries are deleted and treated as misses.
  • Host reports: CheckConsistencyInput.semantic_host_reports accepts the report side of prepare_semantic_continuity. Reports are validated against the current request; stale/duplicate/unknown requests are typed StaleHostRequest errors. Valid host reports are cached with route=host.
  • Shadow/on: shadow runs the candidate/judge/cache path and records would-be warnings/confidence/cache/token/latency metrics without emitting findings. On emits policy-approved semantic issues with the candidate's existing check type and optional evidence. Off performs no candidate/judge work.
  • Budget/latency: per-call 45 s timeout, prompt/output constants, run input-token ceiling, candidate/scene caps already enforced by P2, and typed BudgetExceeded/Timeout/Truncated errors.
  • No migration other than V0058.

Live shadow sample

Run through ai run OLLAMA_API_KEY -- ... against https://ollama.com/v1 with gemma4:31b; no secret was printed. One chapter-1 shadow scene per set:

Set Candidates Model calls Input tokens Output tokens Wall latency Errors Warnings/Info
Baseline 7 1 3,943 971 2,842 ms 0 0 / 0
Held-out #72 7 1 4,225 948 2,754 ms 0 0 / 0

The router/provider did not expose a configured price, so no dollar cost was fabricated; the token and wall-latency figures are the measured per-scene cost inputs. Cache miss = 1 for each sample; a repeat scene with unchanged identity would be a cache hit.

Regression

Semantic mode is off by default. The four deterministic eval sets are unchanged:

Set Result
Baseline 28/28, 3 control findings
Held-out #72 54/56, 0 control
Held-out-2 55/56 localized, 56/56 type-present, 0 control
Held-out-3 26/56 localized, 28/56 type-present, 0 control warnings, 8 world-rule info

Timeline: 4/4, 8/8, 8/8, 3/8. Temporal control warnings: 0.

Tests

  • cargo test -p spindle-adapters --lib semantic_judge::tests: 11 (parser/fencing/evidence, policy, review fallback, rating denial, rating-resolved cache identity, endpoint invalidation, world-rule dedupe controls, attribute subject/predicate/value dedupe).
  • cargo test -p spindle-adapters --test semantic_judge_p3: 5 (host report on/shadow with sections/Markdown, stale report, local-stub fail-closed, cache key/deletion, deterministic-wins dedupe, typed errors).
  • cargo test -p spindle-adapters --test semantic_continuity_p2: 3.
  • Full cargo test --workspace: pass.
  • cargo clippy --workspace --all-targets -- -D warnings: pass.
  • cargo fmt --check: pass.
  • Tool schema, profile-budget, catalog snapshot, and MCP alignment tests: pass.

Held-out-4 was not accessed. CI green, AO approval, threads resolved, checkpoint. Wait for maintainer merge.

Add continuity_semantic/review routing, bounded strict prompt parsing with one repair, typed fail-closed errors, V0058 validated-report cache, host report validation, shadow metrics, and budget/latency guards. Default remains semantic_mode off.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. P3 is off by default, but its enabled path does not yet enforce the specified run budgets or dedupe, the cache can identify the wrong rating-specific judge, and report sections omit appended semantic findings. Required changes are inline. Reviewed 2d22dbb against base 1d14efb. Static inspection only; no tests, builds, or project programs executed.

Enforce run/scene/deadline and prompt/output budgets before judge and repair calls, estimate missing usage, apply deterministic-wins dedupe to emission and shadow counts, key the cache by the rating-resolved binding identity, include scene ids in semantic issues, and build report sections after the semantic stage.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. Final report assembly and scene localization are fixed. Three earlier safeguards remain incomplete: dedupe now suppresses unrelated contradictions, output limiting is not enforced at dispatch and uses cumulative output-minus-input arithmetic, and cache identity still omits the resolved agent/endpoint. Required follow-ups are inline. Reviewed 7ad5335 against base 1d14efb, including the follow-up diff. Static inspection only; tests and project programs were not executed.

Use structured contradiction identity for dedupe, enforce a bounded 1200-token dispatch with prompt reservation and remaining-deadline propagation, and derive cache identity from the resolved agent/provider/endpoint configuration.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. Bounded dispatch, input reservations/deadlines, and resolved agent/endpoint cache identity are now present. Two earlier issues remain: the post-response limit still compares cumulative output against prompt input, and deterministic dedupe is not canon-specific for attribute findings. Required follow-ups are inline. Reviewed daa37f7 against base 1d14efb, including the latest follow-up diff. Static inspection only; tests and project programs were not executed.

Compare per-response output tokens directly against the semantic cap and require matching canonical predicate/value for attribute dedupe. Add attribute and world-rule dedupe control tests.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. The per-call output accounting finding is fixed. One attribute-dedupe identity gap remains: predicate/value text alone conflates different subjects with the same canonical attribute in the same scene. The required follow-up is inline. Reviewed ffbfd23 against base 1d14efb, including the latest diff. Static inspection only; tests and project programs were not executed.

Carry validator entity ids into consistency issues and caches, and require matching subject, predicate, and value before suppressing a semantic attribute finding.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. The attribute validator now produces structured IDs, but the adapter drops them before returning or caching issues, so the prior end-to-end dedupe finding remains unresolved. The required wiring/cache follow-up is inline. Reviewed d444b16 against base 1d14efb, including the latest diff. Static inspection only; tests and project programs were not executed.

Forward validator entity ids into Phase-4 consistency issues and cached details, bump the validator output hash version, and assert attribute subject/scene identity on fresh and cached runs.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. Attribute subject forwarding and cache invalidation are fixed. One related cache-path gap remains for world-rule/voice/reachability findings: cached IDs omit the scene ID that fresh emission adds, changing localization and dedupe on reruns. The required follow-up is inline. Reviewed 8ebe583 against base 1d14efb, including the latest diff. Static inspection only; tests and project programs were not executed.

Merge the scene id into cached Phase-4 issue entity ids so localization and semantic dedupe match fresh emission.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. The remaining cache-path finding is addressed: cached_validator_issues now always merges the scene ID with stored entity IDs, preserving localization and deterministic/semantic dedupe on cache hits. The earlier bounded dispatch, input/deadline guards, resolved judge cache identity, per-response output accounting, attribute-subject forwarding, and final report assembly fixes remain intact. Reviewed 01f2ab4 against base 1d14efb, including the latest follow-up diff. No further required changes found. Static inspection only; tests and project programs were not executed.

@VerifiedOrganic
VerifiedOrganic merged commit 54503f4 into main Nov 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant