feat(96): hierarchy-aware table exactness metric + the calibration that kills the proposed accept rule

[r-uben]

Summary

GH-96 Unit A of the agreed split (metric + fixture first, escalation lane second).
Two deliverables — and the second is a negative result that changes Unit B's design
before any of it is written.

Non-goals: no escalation lane, no engine calls, no config or CLI changes.

1. The existing metric cannot see this failure mode

BenchmarkScorer.score_table_cells compares numeric cells positionally and, on a
shape mismatch, degrades to multiset recall. The GH-96 failure is a
permutation — every digit present, right column, wrong row — so recall is perfect.

On OBR EFO November 2022 page 13:

engine	score_table_cells	new metric
socr strict-local	100.0%	38.4%
escalated (vision)	100.0%	100.0%

It gives the identical 100.0% to a scrambled table and a perfect one. No past
benchmark number is informative about this failure mode.

benchmark/table_exactness.py keys cells to rows identified by label, matched in
document order, so a label reused under two parents (Other measures appears twice
in that table) cannot be credited twice against the same source row. That collision
is why the bake-off reported the escalated page 13 at 95.2%; it is exactly 100.0%.

Ground truth is model-free from the native text layer. Three things it must get
right, each learned from real data rather than assumed:

• Scope to located table regions — otherwise the parser reads prose containing
  numbers ("rising to £61.7 billion (2.1 per cent of GDP)") as table rows.
• Group rows by vertical overlap, not fitz block/line — a PDF table emits the
  label and its numbers as separate text objects, so block/line splits one visual
  row in two. Parameter-free: words on a row have overlapping y-intervals.
• Split at the last non-numeric word — "Growth Plan after 17 October
  reversals" has a numeral inside the label. An x-clustered column boundary is
  chicken-and-egg (that stray 17 forms its own cluster and drags the boundary left);
  this rule is parameter-free and correct for both cases.

ceiling_note marks a degenerate parse so an engine tie is never misread as a
shared failure — but deliberately not when the prediction is empty, since an
engine that emitted no table has genuinely failed. Conflating those hid the three
largest wins in the calibration below on my first pass.

2. The calibration — and it kills the proposed accept rule

The plan gated Unit B on one question: does defect_severity improve on the pages
where exactness improves?

It improves on zero pages. 9 of 9 genuine gains would be rejected.

Two independent causes, both visible in the report:

• Only lane_gap ever moves, and it moves the wrong way. The other four
  severity components are 0 on every page here. On pages 46/48/55 socr emitted
  no table at all, so lane_gap starts at 0 and rises to 8/4/5 once a real
  multi-column table is produced. The metric is anti-correlated with quality:
  emitting nothing scores better than emitting a correct table.
• label_binding_failure is 0 on page 13 — the canonical hierarchical-shift
  page, whose entire defect is label binding. So it fails as the proposed
  trigger as well as the accept test.

This also confirms the design-review finding that severity carries no safety
weight: a well-formed fabrication improves exactly these components.

Consequences for Unit B, recorded in the log note:

• do not use severity improvement as the accept test, or even as a secondary filter;
• do not use label_binding_failure as the trigger;
• the accept test must be the canary alone (incumbent-calibrated token containment)
  plus a well-formedness check drawn from somewhere else;
• a trigger still needs designing from signals that demonstrably fire, and must be
  cross-tabulated the same way before being trusted.

Aggregate, restated

	cells	exact
socr strict-local	2361	1044	44.2%
escalated	2361	2006	85.0%

The bake-off scratch scorer said 38.6% vs 74.0% over 2254 cells. Both rise as its
duplicate-label collision and parser ceiling clear; the gap is essentially
unchanged.

Test plan — observed results

• pytest tests/test_gh96_table_exactness.py -q → 19 passed
• pytest tests/ -q → 1285 passed, 1 xfailed
• ruff format --check + ruff check on both Python files → clean; the log note
  also passes ruff@latest format --check (markdown code blocks, per fix(ci): unblock the format gate — ruff 0.16 formats Markdown code blocks #102)
• Fixtures are generated with fitz at test time — no third-party document content
  enters the repo. Hierarchical / spanning-header tables: nested rows scramble while headline totals look right #96 accepts "OBR-style … or redacted minimal".

Includes test_the_positional_scorer_cannot_see_what_this_one_measures, which pins
the justification: the positional scorer scores the permutation 1.0 while this one
scores it under 50%.

Reviewer notes

• Fully model-free and hermetic — no engine, no network, no provider ladder.
• Highest-risk logic is the native ground-truth parser; the three rules above are
  each covered by a dedicated test.
• The calibration is reproducible from the paths in the bake-off note's Setup table.

Refs #96