Skip to content

fix(semantic): remove control false positives and repair inflation - #104

Merged
VerifiedOrganic merged 6 commits into
mainfrom
ao/planning-agent-entry/semantic-precision
Nov 12, 2026
Merged

VerifiedOrganic merged 6 commits into
mainfrom
ao/planning-agent-entry/semantic-precision

Conversation

@VerifiedOrganic

@VerifiedOrganic VerifiedOrganic commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

What this round fixes

Precision round for the semantic continuity lane (still off by default). This PR:

  • removes the control false positives on baseline (+2 quantity) and held-out-2 (+1 dead-character acting);
  • brings the on-mode repair rate to <=10% per call (1.19-3.80% measured);
  • fixes the shadow/on parity mismatch on feat(evals): held-out continuity contradiction corpus #72 by measuring parity from actual shadow and on passes;
  • addresses the three AO review findings on d1925528.

Review findings

  1. UTF-8 context windows (semantic_judge.rs): the ±240-byte dead-context window now rounds both edges to character boundaries; regressions cover multibyte characters crossing the left and right edges.
  2. Judge-policy cache revision (semantic_judge.rs): SEMANTIC_JUDGE_POLICY_VERSION is now semantic-judge-policy-v3; cache-key construction is shared, and a test seeds a v2 full report (which still parses as a contradiction) and proves a v3 lookup misses it.
  3. Shadow parity from actual results (semantic_eval.py): the evaluator runs a fresh on pass, then a real cache-only shadow pass, keeps each mode's own results, and computes parity from the two actual summaries. No derived shadow view is used as evidence.

General precision fixes

  • Quantity contradictions require balance/stock evidence, not a price or transaction.
  • Dead-character state uses only states strictly before the scene; memory/flashback/vision framing is rejected.
  • Deterministic dedupe now covers same-subject/same-scene findings for families beyond world rules (this removed the baseline dead-character duplicates); secret dedupe also requires the same secret fact, and the remaining families require their own canon/event/fact identity, so an unrelated scene-level finding cannot suppress an independent conflict.
  • Family evidence gates run on compact, full, and host reports before caching or emission, so the supported report forms cannot bypass the precision policy.
  • Secret disclosures require a distinctive value detail or a public-audience marker, so a private accusation that only names the topic is not a leak.
  • No set-specific rules; all gates are family/evidence/general.

Results

Judge ollama:gemma4:31b, temperature 0, seed 7; same config for all sets. on is the fresh dispatch pass; shadow reuses the validated-report cache.

Set Deterministic Semantic on Warning precision Control warnings On repair Shadow/on parity Anchor validity p95 prompt/output/latency
Baseline 28/28 28/28 n/a (0 warnings) 0 1.18% (85 calls) pass 1.00 2,173 / 627 / 1,797 ms
#72 54/56 54/56 1.00 0 2.63% (76 calls) pass 1.00 2,893 / 1,036 / 2,507 ms
Held-out-2 55/56, 56/56 type 55/56, 56/56 type n/a (0 warnings) 0 0.00% (66 calls) pass 1.00 2,029 / 646 / 1,734 ms
Held-out-3 26/56, 28/56 type 29/56, 30/56 type 1.00 0 4.17% (72 calls) pass 1.00 3,216 / 951 / 2,466 ms

Control warnings do not rise on any set. Warning precision is 1.0 where warnings exist. Evidence-anchor validity is 1.0. On-mode repair is <=4.17% of calls. Shadow/on would-be finding counts match on every set (the cache-only shadow pass makes only 0-6 retry dispatches, so its own repair ratio is reported separately in the checked-in reports and is not the gate).

Gates

  • Deterministic --semantic off unchanged: baseline 28/28 with 3 control findings; feat(evals): held-out continuity contradiction corpus #72 54/56 with 0; held-out-2 55/56 + 56/56 type with 0; held-out-3 26/56 + 28/56 type with 0 control warnings.
  • No control-warning rise; warning precision >=0.85 (1.0 where defined); on repair <=10% per call; anchor validity >=98%; one repair per call.
  • cargo fmt --all -- --check, cargo clippy -p spindle-core -p spindle-adapters --all-targets -- -D warnings, cargo test -p spindle-adapters --lib (900 passed), cargo test -p spindle-core --lib (358 passed).
  • Default remains semantic_mode = off; activation gates still not met because aggregate semantic lift is +3 localized/+2 type-present (all held-out-3).
  • Held-out-4 untouched; no sealed-corpus data was read.

Checked-in reports: evals/results/semantic_precision_*.json.

Require balance evidence for quantity drift, strictly-prior dead state with memory framing rejection, and a shared on/shadow execution to remove repair inflation and parity drift. Control warnings are eliminated; remaining gap is recall.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. Required fixes: make death-context slicing UTF-8 safe, invalidate cached reports produced before the new policy filters, and measure shadow parity/cost from actual shadow results rather than a copy of on-mode output. Reviewed d192552 against ed1a1d5. Static review only; no tests or project programs were run and the checkout was not modified.

Comment thread evals/semantic_eval.py Resolved

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. Two correctness issues remain: the new evidence gates are bypassed by full/host reports, and deterministic dedupe can suppress distinct canon conflicts. Static review against origin/main; no programs, builds, or tests were run.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested. The full/host evidence-gate fix and secret-specific identity check address the concrete cases from the previous review. The general dedupe fallback still suppresses distinct canon conflicts in other families; one required inline finding describes the remaining issue. Static review against origin/main, including the follow-up diff; no programs, builds, or tests were run.

@VerifiedOrganic VerifiedOrganic left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — ready. The previously reported issues are addressed: shared evidence gates cover compact, full, and host reports, and deterministic dedupe now requires canon identity for independent fact/event families while preserving character-scoped death dedupe. The added regressions cover unrelated subjectless timeline candidates and multiple knowledge facts for the same character. No further high-confidence findings requiring changes. Static review of the diff against origin/main and the follow-up changes at ff8626f; no programs, builds, or tests were run.

@VerifiedOrganic
VerifiedOrganic merged commit 2212d75 into main Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant