docs(96): calibrate the escalation trigger — native-exactness dominates dualpass_flagged - #106
Merged
Merged
Conversation
…es dualpass_flagged The calibration section closed by saying the Unit B trigger still needed designing from signals that demonstrably fire. Cross-tabulating every candidate against the ground truth of whether escalation actually helps a page: native-exactness < 100% 13/16 fires, 69% precision, 100% recall a GT row is missing 13/16 fires, 69% precision, 100% recall orphan value rows 4/16 fires, 100% precision, 44% recall tables declared, none emitted 3/16 fires, 100% precision, 33% recall dualpass_flagged (current) 16/16 fires, 56% precision, 100% recall A candidate the original sketch never considered dominates: on a born-digital page the exactness metric is computable at runtime, model-free and free, because the native text layer is right there. It is a direct measure of the defect rather than a structural proxy. It strictly dominates what socr flags on today — identical recall, better precision, three fewer pages. The discrimination is exactly what #95 found missing: pages 45, 59 and 60 are 100% correct and this trigger stays silent on all three, while dualpass_flagged fires on them because it fires on every table page. Parameter-free ("disagrees at all", not a tuned threshold), and fail-safe in direction: over-triggering costs ~$0.0002 per wasted call and escalation was never worse than the incumbent on any page. Limits recorded: born-digital only; one document with no negative controls, so 69% is a point estimate and only the ordering is robust; the trigger works on the mode-(a) pages because the shift also drops rows, not because it detects misalignment as such; page 53 is a permanent false positive until its native parse is fixed. Refs #96
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Docs only. Closes the open design question left by the Unit A calibration (#104),
which invalidated both load-bearing pieces of the Unit B sketch and said a trigger
"still needs designing from signals that demonstrably fire".
Every candidate is now cross-tabulated against the ground truth of does escalation
actually help this page.
dualpass_flagged(current)Recommendation: escalate when the emitted table disagrees with its own native layer
A candidate the original sketch never considered turns out to dominate. On a
born-digital page the exactness metric is computable at runtime, model-free, at
zero cost — the native text layer is right there. That is a direct measure of the
defect rather than a structural proxy for it.
It strictly dominates the signal socr flags on today: identical recall (no
missed gains), better precision, three fewer pages fired.
The discrimination is precisely what #95 found missing. Pages 45, 59 and 60 are
100% correct and this trigger stays silent on all three — while
dualpass_flaggedfires on them, because it fires on every table page in the document.
Properties that matter for the lane:
exactness < 100%and
rows_not_found > 0select the same 13 pages.escalation was never worse than the incumbent on any page. Over-triggering costs
money, not correctness.
is a poor oracle the trigger fires and the canary then rejects, so the page is
left alone at the cost of a wasted call.
Limits, recorded in the note
canary can run.
from one report; only the ordering against
dualpass_flaggedis robust.the shift also drops rows (parents emptied, orphans trailing). A permutation that
preserved cell-to-label binding would be invisible to both trigger and canary —
not a failure mode observed here, but worth knowing.
precision is 69% rather than 75%.
Test plan
Not applicable — documentation. Reproducible from the paths in the bake-off note's
Setup table plus
benchmark/table_exactness.score_page, which shipped in #104.Refs #96