Support line-level PDF observations when GPO margin line numbers are absent

[willhea]

Type: architectural research spike (upstream of heading classification)
Status: research spike — external to this repository and read-only against the checkout until falsification completes
Related: #524, #501, #54 (leveled heading tree), ADR 0019 (observation identity), ADR 0020

---

Problem

DeltaTrack attaches glyph and geometry evidence to a printed line through the GPO margin
line number. parsers/pdf_text._page_glyph_sizes reconstructs each visual line from the
glyph walk, matches it against _NUMBERED_LINE (^(\d{1,2}) (.*)$), and keys the resulting
(glyph_size, LineGeom) sidecar by that number. _attach_geometry then joins the sidecar
back onto merged Lines by the same key. A line with no margin number gets
glyph_size=None and geom=None.

Enrolled bills largely do not carry margin line numbers. Measured on the committed corpus:
118-hr-4366/6_enrolled-bill yields 68 numbered lines across 428 pages. With almost no
sized lines, pdf_anchors.derive_size_bands returns None, and the account level is never
produced for that document at all.

Consequence, measured: of the 52 committed PDF/XML-twin versions, 10 enrolled versions
produce zero heading boundaries under the current pipeline. Independent XML/raw-line
seeding found 4,457 oracle-resolvable WRAP/STACK boundaries in those same 10 versions.
The evidence is present in the PDF; it is dropped before structural parsing because there is
no attachment key.

This is an attachment-correspondence limitation, not a heading-rule limitation. It is
recorded separately from #524/#501, whose numbered-margin ambiguity is a different and
independently closed question.

Research question

Can DeltaTrack establish a deterministic one-to-one correspondence between raw PDF
visual-line observations and the corresponding pre-merge cleaned print_lines, without
relying on GPO margin line numbers, while preserving the existing print_line → merged
Line semantics and numbered-margin behaviour?

This is deliberately not a proposal for a particular key. Baseline y, character origin,
x extent, reconstructed line text, PDF object/show ordering and combinations of these are
candidate correspondence evidence to be tested and falsified, not a frozen design.

Terminology: this work is about attachment correspondence between two representations of
the same printed material. ADR 0019 already defines observation identity, and this brief
does not redefine it.

First research task

Determine whether a deterministic attachment correspondence exists and try to break it
against four attacks:

1. same-baseline collisions,
2. multi-print-line merges,
3. cross-process and object/show-order perturbation,
4. numbered-margin equivalence.

If the correspondence survives all four, the attachment contract can be frozen, and the
consequences of enabling the existing anchor machinery on enrolled bills are reviewed
separately. If it does not survive, that is the result.

---

Execution boundary

This spike is research-first and external to the DeltaTrack repository.

Until a candidate attachment correspondence has survived F1–F3 and the numbered-margin
preservation comparison (F4):

• treat the DeltaTrack checkout as read-only;
• keep prototype code, generated PDFs, extracted observations, comparison artifacts and
  results in the external research workspace;
• do not modify pdf_text.py;
• do not move pdf_parser_revision();
• do not regenerate ADR 0019 traces, goldens or canonical baselines;
• do not create an implementation branch or PR.

The candidate mechanism may import DeltaTrack or reproduce its extraction functions in an
external harness. Numbered-margin equivalence should be tested as a shadow comparison
against current develop, including the deliberate-misattachment negative control.

What a surviving mechanism returns

If a correspondence mechanism survives the spike, return:

• the proposed attachment contract;
• results of F1–F3;
• the numbered-margin equivalence result and its negative control;
• the downstream activation census.

Only after separate review may production implementation in pdf_text.py be authorized.
That later implementation is what moves the ADR 0019 parser revision and triggers any
quarantine or regeneration work.

---

Falsifiers the spike must run

F1 — Same-baseline collision

The existing glyph walk clusters characters by vertical position alone before sorting by
x: _cluster_baselines sorts on the char-box bottom (sorted(chars, key=lambda c: -c[0]))
with tolerance 0.5 × median_size and never consults x; _line_text then sorts the cluster
by left x and inserts positional spaces. Two spatially separate printed text runs sharing a
baseline therefore merge into one interleaved "line". That is a concrete failure mode for any
attachment resting on vertical position.

Requirement. Find in the corpus, or construct, a page containing two spatially separate
printed text runs sharing the same baseline y. The candidate correspondence must keep them
distinct. Whether such pages exist in the committed corpus is itself part of this task —
do not assume either way.

Negative control. Reverse or otherwise perturb their PDF object/show ordering. The
correct cleaned line must still receive the correct observation.

Rejection criterion. A mechanism that collapses same-y columns or table cells is
rejected, however well it scores elsewhere.

Prior measurement bearing on this: the bucketing granularity is already known to be
load-bearing in both directions. In the external study, joining content-stream records to
extracted lines produced 383 mismatches at a 2-decimal y bucket versus 18 at an integer
bucket, because a caps-and-small-caps heading's large initial capital sits ~0.01 pt from its
small caps and must not be split. A tolerance wide enough to hold one line together is
moving toward the tolerance that would fuse two adjacent columns.

F2 — Printed-line → merged-line ownership

DeltaTrack does not stop at physical printed lines:

raw PDF → _parse_print_lines() → print_lines → _merge_print_lines() → merged Line

Page.merge_ranges already records the [start, end) slice of print_lines each merged
line was built from. The current semantics, which the spike must first reproduce rather
than redesign:

• _merge_print_lines joins WORD- + lowercase continuation across printed lines, and the
  merged line is constructed as Line(current.line_number, …) — it inherits the first
  printed line's number;
• _attach_geometry therefore looks up the sidecar for that first printed line only, so the
  continuation lines' glyph size and geometry are not represented in the merged line.

Requirement. Determine explicitly how attached observations behave when one merged
Line spans multiple physical printed lines. Do not invent new semantics during the spike;
first require the candidate machinery to reproduce today's numbered-path behaviour exactly.

Falsifier. An unnumbered soft-hyphen / multi-print-line case. A correspondence
mechanism must not pass merely because ordinary headings happen to occupy a single physical
line — that is the easy case and it is not the one that decides this.

F3 — Ordering and cross-process determinism

PDFium's reading order is scrambled for these documents, so any ordering the correspondence
relies on must be derived from geometry rather than from emission order. ADR 0019 leaves
PDF emission determinism measured in-process only; if ordering becomes load-bearing for
attachment, that open question becomes load-bearing too and must be closed here.

Falsifier. Perturb object/show ordering and re-run across processes; the correspondence
must be invariant.

F4 — Numbered-margin equivalence

An eventual production implementation touching pdf_text.py will necessarily move
pdf_parser_revision() under ADR 0019. The external research spike should measure the
expected consequences without causing that revision move itself. Because that move is
unavoidable at implementation time, literal byte equality of the full emitted output is
impossible as an invariant and must not be used as one.

Requirement. Numbered-margin parsing remains semantically identical. Over the
numbered-margin corpus, compare a frozen projection of result-bearing parser output that
excludes only the expected parser_revision / observation-identity change. Line text,
line numbering, attached glyph size and geometry, anchor structure, and downstream canonical
results must otherwise be unchanged. Any non-identity delta is a finding requiring
adjudication, not something to absorb into a regenerated baseline.

Negative control. Deliberately misattach one numbered line's sidecar and prove the
equivalence harness goes red. Existing goldens being green is not sufficient evidence
that the comparison can fail.

---

Extraction cost: test the cheaper path first

Do not assume FPDFText_GetCharOrigin is required. The glyph walk already retrieves each
character's box via _char_box and already uses the box bottom as its clustering key
(chars.append((bottom, left, right, cp, size))), discarding it only at the line record —
LineGeom keeps three x-coordinates and no y.

Frame the extraction decision as a comparison, in this order:

1. Can the vertical geometry already available in the existing glyph walk support
   deterministic attachment?
2. Only if (1) fails: does FPDFText_GetCharOrigin materially improve correctness or
   stability enough to justify another FFI call and a retained observation?

Prefer the existing evidence where it suffices. Do not widen the extraction contract before
the cheaper path has been falsified.

---

Downstream activation census — record, do not bless

Better attachment will switch on machinery that is currently inert on enrolled bills. The
spike must measure that, as a consequence census:

• whether derive_size_bands() begins returning bands;
• newly emitted anchors;
• changes in observation counts and types;
• canonical diff changes;
• any ADR 0019 trace / quarantine effects.

This is a census, not authorization to change heading classification. If downstream
structural output changes, report each result-bearing delta and stop short of accepting it
as correct behaviour. Whether that output is right is a separate review.

---

Invariants

1. Correspondence uniqueness. Where attachment is claimed, each admitted raw PDF visual
   line maps deterministically to the correct pre-merge cleaned print_line, and each
   such print_line receives at most one admitted visual-line observation; spatially
   separate runs sharing a baseline do not collapse.
2. Merge ownership. The existing print_line → merged Line relationship and
   first-line observation ownership are preserved before any redesign is considered.
3. Backward equivalence. Numbered-margin semantic output is unchanged modulo the
   unavoidable ADR 0019 parser-revision / identity change, with a mutation proving the
   comparison can fail.
4. No structural-policy smuggling. The spike measures downstream activation caused by
   better attachment but introduces and approves no heading segmentation or classification
   rule.

---

Scope

In scope: establishing and falsifying a deterministic attachment correspondence for
unnumbered layouts; attaching the existing glyph/geometry observations through it; preserving
numbered-margin behaviour; censusing downstream activation.

Out of scope: any heading segmentation or wrap/stack policy; any change to #524/#501
behaviour; any new heading feature; re-opening #662; redefining ADR 0019 observation identity.

Other design tensions to resolve inside the spike, not before

• Cache identity. tests/pdf_corpus._extractor_fingerprint keys on pdf_text.py source,
  so it should invalidate automatically — confirm rather than assume.
• ADR 0019 quarantine. The revision move is authorized by design but must be planned and
  its trace effects measured, not discovered late.

Evidence already available (external, not in the repository)

~/Documents/DeltaTrack-research/pdf-observation-surface/ holds the 4,457-boundary enrolled
population with XML-derived labels, the 2,779-boundary numbered-margin population, the
ordered/local XML oracle and its locality controls, the extraction liveness controls, and the
reconciled 52-version ledger. A spike can reuse the populations directly rather than
re-deriving them.

Motivating measurement, recorded for context only: on the 4,457 enrolled boundaries,
baseline spacing separates STACK (19.20–23.20 pt, n=3,665) from WRAP (9.90–10.40 pt, n=792)
by 8.80 pt, consistent across 9/9 documents carrying both classes. That is why attachment is
worth solving. It is not a mandate to build a classifier, and no segmentation policy is
decided here.