docs: OBR hierarchical-table bake-off — escalation viability and a fabrication null result - #101
Merged
Merged
Conversation
r-uben
force-pushed
the
docs/97-obr-table-bakeoff
branch
from
July 30, 2026 14:21
85aa499 to
70ef9b0
Compare
… escalation viability, fabrication null result Measures the 19 flagged pages of an existing strict-local OBR EFO run against the PDF's native text layer (born-digital, so ground truth is free): 870/2306 cells exact = 37.7%, across four distinct failure modes rather than the single hierarchical-shift mode #96 describes. Answers #96's escalate-vs-fail-closed fork empirically: agy on the page image scores ~100% on the worst page vs 32.5% strict-local, so escalation is viable. But a sandboxed run that could not read the image returned confident, entirely fabricated data at exit 0 which would have passed every model-free structural heuristic — so any escalation lane needs a grounding canary and provenance, not a quality check. Confirms #95's premise: the dual-pass detector had zero false negatives (thin base, one negative control), so the withheld signal is reliable. Refs #95, #96, #97.
…sification Escalation run extended from 1 page to all 19 with the canary gate: socr 870/2254 = 38.6% vs agy 1668/2254 = 74.0%. agy was never worse on any page, and recovered page 46 (socr fail-closed, stub + image) and page 55 (socr dropped the table) exactly against the native layer. Mode (b) label shredding is fixed too, so #96's scope was too narrow. Zero canary mismatches across 19 correctly configured runs. Correction: pages 37/41/43 were wrongly listed as silently dropped tables. They contain Chart 16, Chart 18 and Chart D; the ground-truth parser mistook chart data labels for table rows. socr was correct to emit no table there and the escalation run independently returned NO TABLE. Aggregate recomputed over the 16 real table pages. Also documents a second scorer limitation: the native-layer parser caps some pages well below 100%, so an engine tie on this metric usually means the ceiling was hit, not that both failed. Refs #95, #96, #97.
…ng correct ones Validating the #95 trust derivation against the real audit log broke Result 2's claim. Page 66 IS flagged, by value_guard_row_count_warning (pages 13, 46, 48, 51, 55, 64, 65, 66, 67). The earlier union only covered dualpass_flagged, native_table_verifier_warn, table_region_unverifiable and page_failed. Consequence: 20 of the 20 substantial-table pages are flagged, so there are no negative controls and "zero false negatives" was untestable, not a finding. Recall is trivially 100%; precision cannot be measured and is visibly poor — page 45 scores 100.0% and is flagged, page 67 94.1%, page 66 91.2%. Reframes #95: the sidecar is a triage list ("look at these"), not a trust verdict ("these are wrong"). Still worth surfacing over a metadata.json that names one page, but severity ranking becomes a real follow-up since the signal is binary and saturated. Also moves page 48 from mode (b) label shredding to mode (d) repetition runaway: it is a wholly contentless 12-column table (empty header, separator, 15 identical empty rows), so nothing was shredded — it was never populated. Records the #97 root cause found while fixing it: the pre-existing guard had an undocumented 20-char floor AND ran at the engine boundary, before table assembly.
CI resolves ruff latest (0.16.0) rather than uv.lock's 0.15.17, and 0.16 formats Python code blocks inside Markdown by default where 0.15 gated it behind --preview. The appendix scorer was hand-written, so it failed the repo-wide format check.
r-uben
force-pushed
the
docs/97-obr-table-bakeoff
branch
from
July 30, 2026 14:52
70ef9b0 to
3748ed5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Decision record for the OBR hierarchical-table bake-off: an empirical pass over an
existing strict-local run that answers #96's escalate-vs-fail-closed design fork
with a measurement instead of a panel, and checks #95's premise.
Docs only — no code.
Findings
image 74.0% label-keyed cell exactness across 16 table pages. Never worse on
any page. Recovers page 46 (socr fail-closed with a stub + screenshot) and page
55 (socr silently dropped the table) exactly against the native layer. So
fail-closed should be the fallback after escalation, not instead of it.
the image returned 741 bytes of confident, well-formed, entirely invented
fiscal data at exit 0 with empty stderr — and it satisfied every structural
instruction, so it would have scored better than the real 32.5% output on every
model-free heuristic. Escalation therefore needs a grounding canary, not a
quality check. The native text layer supplies one free on born-digital pages.
negative controls, and page 45 scores 100.0% while flagged.
(parent, label)path — hierarchical tables reuse child labels.Corrections carried in the history
The note records three of my own errors rather than quietly fixing them, because
each changes how the numbers read:
Chart 16, Chart 18, Chart D; the ground-truth parser mistook chart data
labels for table rows. socr was correct to emit no table, and the escalation run
independently returned
NO TABLE. Aggregate recomputed over the 16 real tablepages.
flagged by a kind the earlier union missed.
contentless table.
The scorer is included inline with both its limitations documented (duplicate-
label collision; a per-page ceiling that makes an engine tie mean "ceiling hit",
not "both failed").
Test plan
Not applicable — documentation. Every number in the note is reproducible from the
paths named in its Setup table plus the scorer in its appendix.
Related