Skip to content

docs(96): calibrate the escalation trigger — native-exactness dominates dualpass_flagged - #106

Merged
r-uben merged 1 commit into
mainfrom
docs/96-trigger-calibration
Jul 30, 2026
Merged

docs(96): calibrate the escalation trigger — native-exactness dominates dualpass_flagged#106
r-uben merged 1 commit into
mainfrom
docs/96-trigger-calibration

Conversation

@r-uben

@r-uben r-uben commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

Docs only. Closes the open design question left by the Unit A calibration (#104),
which invalidated both load-bearing pieces of the Unit B sketch and said a trigger
"still needs designing from signals that demonstrably fire".

Every candidate is now cross-tabulated against the ground truth of does escalation
actually help this page
.

trigger fires TP FP missed precision recall
native-exactness < 100% 13/16 9 4 0 69% 100%
a ground-truth row is missing 13/16 9 4 0 69% 100%
orphan value rows present 4/16 4 0 5 100% 44%
tables declared, none emitted 3/16 3 0 6 100% 33%
dualpass_flagged (current) 16/16 9 7 0 56% 100%

Recommendation: escalate when the emitted table disagrees with its own native layer

A candidate the original sketch never considered turns out to dominate. On a
born-digital page the exactness metric is computable at runtime, model-free, at
zero cost
— the native text layer is right there. That is a direct measure of the
defect rather than a structural proxy for it.

It strictly dominates the signal socr flags on today: identical recall (no
missed gains), better precision, three fewer pages fired.

The discrimination is precisely what #95 found missing. Pages 45, 59 and 60 are
100% correct and this trigger stays silent on all three — while dualpass_flagged
fires on them, because it fires on every table page in the document.

Properties that matter for the lane:

  • Parameter-free — "disagrees at all", not a tuned threshold. exactness < 100%
    and rows_not_found > 0 select the same 13 pages.
  • Model-free and free — no engine call to decide whether to make an engine call.
  • Fail-safe in direction — the four false positives cost ~$0.0002 each, and
    escalation was never worse than the incumbent on any page. Over-triggering costs
    money, not correctness.
  • Same oracle as the canary, and it degrades coherently: where the native layer
    is a poor oracle the trigger fires and the canary then rejects, so the page is
    left alone at the cost of a wasted call.

Limits, recorded in the note

  • Born-digital only. A scanned page has no native oracle, so neither trigger nor
    canary can run.
  • One document, no negative controls. 69% is a point estimate on 16 table pages
    from one report; only the ordering against dualpass_flagged is robust.
  • It does not detect misalignment as such. It catches the mode-(a) pages because
    the shift also drops rows (parents emptied, orphans trailing). A permutation that
    preserved cell-to-label binding would be invisible to both trigger and canary —
    not a failure mode observed here, but worth knowing.
  • Page 53 is a permanent false positive until its native parse is fixed, and is why
    precision is 69% rather than 75%.

Test plan

Not applicable — documentation. Reproducible from the paths in the bake-off note's
Setup table plus benchmark/table_exactness.score_page, which shipped in #104.

Refs #96

…es dualpass_flagged

The calibration section closed by saying the Unit B trigger still needed designing
from signals that demonstrably fire. Cross-tabulating every candidate against the
ground truth of whether escalation actually helps a page:

  native-exactness < 100%     13/16 fires, 69% precision, 100% recall
  a GT row is missing         13/16 fires, 69% precision, 100% recall
  orphan value rows            4/16 fires, 100% precision, 44% recall
  tables declared, none emitted 3/16 fires, 100% precision, 33% recall
  dualpass_flagged (current)  16/16 fires, 56% precision, 100% recall

A candidate the original sketch never considered dominates: on a born-digital page
the exactness metric is computable at runtime, model-free and free, because the
native text layer is right there. It is a direct measure of the defect rather than
a structural proxy.

It strictly dominates what socr flags on today — identical recall, better precision,
three fewer pages. The discrimination is exactly what #95 found missing: pages 45,
59 and 60 are 100% correct and this trigger stays silent on all three, while
dualpass_flagged fires on them because it fires on every table page.

Parameter-free ("disagrees at all", not a tuned threshold), and fail-safe in
direction: over-triggering costs ~$0.0002 per wasted call and escalation was never
worse than the incumbent on any page.

Limits recorded: born-digital only; one document with no negative controls, so 69%
is a point estimate and only the ordering is robust; the trigger works on the
mode-(a) pages because the shift also drops rows, not because it detects
misalignment as such; page 53 is a permanent false positive until its native parse
is fixed.

Refs #96
@r-uben
r-uben merged commit 5729819 into main Jul 30, 2026
3 checks passed
@r-uben
r-uben deleted the docs/96-trigger-calibration branch July 30, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant