From e9a87c5ebae64c3d6a83be2f0f2dd275614e7163 Mon Sep 17 00:00:00 2001 From: Will Hea <1123070+willhea@users.noreply.github.com> Date: Fri, 21 Aug 2026 17:31:12 -0400 Subject: [PATCH 1/8] fix(pdf): rejoin every printed word break, deciding the hyphen from the document (#650) A word GPO's printer broke across a line reached `full_text` broken whenever the continuation was uppercase, so the Title I heading of 118-hr-8752 exported as `INTEL-` / `LIGENCE` and its governing statute as `McKinney-` / `Vento`. The merge required `.islower()` on the continuation, so it declined to join at all. Swept over the fixture corpus with GPO's XML as an oracle, the defect is larger than the uppercase case it was filed for, and in two directions the issue did not name: 1,031 uppercase continuation, never joined -- the filed defect 423 lowercase continuation, joined but WELDED -- `project-based` -> `projectbased` 1,542 break at a PAGE SEAM, never joined -- `_merge_print_lines` runs per page Whether the hyphen belongs to the word is not decidable at the break: GPO breaks a compound at its own hyphen and PDFium marks that identically to a syllable break (`McKinney22 Vento`). It IS decidable from the rest of the document, which almost always spells the word out somewhere it did not have to break. `BreakEvidence` indexes those spellings and decides 47,875 of 50,679 breaks with zero errors against the XML; pooling the compared pair's two documents adds 826 more, also with zero; the remainder falls to letter case. Measured paired on the 50,679 breaks the XML can decide: 2,980 wrong -> 118 wrong, 1,336 fixed, no site regressed. Through the pipeline's own output the corpus goes from 3,201 defective sites to 213, and the 180 that remain are enumerated in tests/data/pdf/word_break_residuals.json with the reason each could not be decided. Not gated on PDFium's U+FFFE marker, though it names the same breaks where it appears. Swept corpus-wide, 1,390 of 52,044 line-final hyphens carry no marker -- every one in an enrolled print, plus a 0-4% tail of ordinary syllable breaks in numbered prints -- so gating on it would stop joining all of them. ADR 0020 evidence for the correspondence that moved: of 23 pinned pairs, 6 changed and all 6 emit FEWER changes (-32 total, almost entirely spurious `added`), none more. That is the #31 defect class -- a word split in one version and whole in the other reading as a change -- clearing. One round-1 revocation moved with it, in 115-hr-5895/3->4 (72 -> 71); the other 34 pairs are identical under both extractors. The gate is hyphen-sensitive, which nothing else in the suite is: `normalize_for_cross_format` deletes hyphens before matching and says so, and `normalize_for_recall` rewrites `Child- Rescue` back into `Child-Rescue` at compare time. Both are right for a recall question and both make this defect invisible. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 1 + examples/hr8752_pdf_diff.html | 9 +- scripts/regen_word_break_residuals.py | 70 ++ src/deltatrack/compare/pdf.py | 25 +- src/deltatrack/parsers/pdf_text.py | 278 ++++- tests/data/pdf/extraction_golden.json | 4 +- tests/data/pdf/word_break_residuals.json | 1264 ++++++++++++++++++++++ tests/data/pdf_canonical_baseline.json | 84 +- tests/data/round1_pairing_sentinel.json | 54 +- tests/test_pdf_round1_revocation.py | 21 +- tests/test_pdf_word_break_recall.py | 260 +++++ web/webapp/sample/example.html | 9 +- 12 files changed, 1966 insertions(+), 113 deletions(-) create mode 100644 scripts/regen_word_break_residuals.py create mode 100644 tests/data/pdf/word_break_residuals.json create mode 100644 tests/test_pdf_word_break_recall.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 676151b9..5e22d9ce 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -253,6 +253,7 @@ jobs: tests/test_pipeline_parity.py tests/test_pdf_xml_amount_recall.py tests/test_pdf_xml_prose_recall.py + tests/test_pdf_word_break_recall.py tests/test_front_matter_parity.py tests/test_xml_compare.py tests/test_toc_tree.py diff --git a/examples/hr8752_pdf_diff.html b/examples/hr8752_pdf_diff.html index d480d7f1..d7dc90df 100644 --- a/examples/hr8752_pdf_diff.html +++ b/examples/hr8752_pdf_diff.html @@ -456,7 +456,7 @@

TITLE II > SECURITY, ENFORCEMENT, AND INVESTIGATIONS > U.S. CUSTOMS AN

TITLE II > SECURITY, ENFORCEMENT, AND INVESTIGATIONS > U.S. IMMIGRATION AND CUSTOMS ENFORCEMENT > OPERATIONS AND SUPPORT

p.13 L7 – p.14 L20p.13 L21 – p.15 L11
-
For necessary expenses of U.S. Immigration and Customs Enforcement for operations and support, including the purchase and lease of up to 3,790 (2,350 for replacement only) police-type vehicles; overseas vetted units; and maintenance, minor construction, and minor leasehold improvements at owned and leased facilities; $10,497,243,000; $10,497,243,000 (increased by $4,000,000) (increased by $2,000,000); of which not less than $6,000,000 (increased by $4,000,000) shall remain available until expended for efforts to enforce laws against forced child labor; of which $46,696,000 shall remain available until September 30, 2026; of which not less than $2,000,000 (increased by $2,000,000) is for paid apprenticeships for participants in the Human Exploitation Rescue Operative Child- Rescue Child-Rescue Corps; of which not less than $15,000,000 shall be available for investigation of intellectual property rights violations, including operation of the National Intellectual Property Rights Coordination Center; and of which not less than $5,900,389,000 shall be for enforcement, detention, and removal operations, including transportation of unaccompanied alien minors, of which not less than $3,081,725,000 shall remain available until September 30, 2026: Provided, That not to exceed $11,475 shall be for official reception and representation expenses: Provided further, That not to exceed $10,000,000 shall be available until expended for conducting special operations under section 3131 of the Customs Enforcement Act of 1986 (19 U.S.C. 2081): Provided further, That not to exceed $2,000,000 shall be for awards of compensation to informants, to be accounted for solely under the certificate of the Secretary of Homeland Security: Provided further, That not to exceed $11,216,000 shall be available to fund or reimburse other Federal agencies for the costs associated with the care, maintenance, and repatriation of smuggled aliens unlawfully present in the United States: Provided further, That not less than $2,000,000 shall be for entering into new agreements for the delegation of law enforcement authority provided by section 287(g) of the Immigration and Nationality Act: Provided further, That funding made available under this heading shall maintain a level of not less than 50,000 detention beds.
+
For necessary expenses of U.S. Immigration and Customs Enforcement for operations and support, including the purchase and lease of up to 3,790 (2,350 for replacement only) police-type vehicles; overseas vetted units; and maintenance, minor construction, and minor leasehold improvements at owned and leased facilities; $10,497,243,000; $10,497,243,000 (increased by $4,000,000) (increased by $2,000,000); of which not less than $6,000,000 (increased by $4,000,000) shall remain available until expended for efforts to enforce laws against forced child labor; of which $46,696,000 shall remain available until September 30, 2026; of which not less than $2,000,000 (increased by $2,000,000) is for paid apprenticeships for participants in the Human Exploitation Rescue Operative Child-Rescue Corps; of which not less than $15,000,000 shall be available for investigation of intellectual property rights violations, including operation of the National Intellectual Property Rights Coordination Center; and of which not less than $5,900,389,000 shall be for enforcement, detention, and removal operations, including transportation of unaccompanied alien minors, of which not less than $3,081,725,000 shall remain available until September 30, 2026: Provided, That not to exceed $11,475 shall be for official reception and representation expenses: Provided further, That not to exceed $10,000,000 shall be available until expended for conducting special operations under section 3131 of the Customs Enforcement Act of 1986 (19 U.S.C. 2081): Provided further, That not to exceed $2,000,000 shall be for awards of compensation to informants, to be accounted for solely under the certificate of the Secretary of Homeland Security: Provided further, That not to exceed $11,216,000 shall be available to fund or reimburse other Federal agencies for the costs associated with the care, maintenance, and repatriation of smuggled aliens unlawfully present in the United States: Provided further, That not less than $2,000,000 shall be for entering into new agreements for the delegation of law enforcement authority provided by section 287(g) of the Immigration and Nationality Act: Provided further, That funding made available under this heading shall maintain a level of not less than 50,000 detention beds.
@@ -560,7 +560,7 @@

TITLE III > PROTECTION, PREPAREDNESS, RESPONSE, AND RECOVERY > FEDERAL

TITLE III > PROTECTION, PREPAREDNESS, RESPONSE, AND RECOVERY > FEDERAL EMERGENCY MANAGEMENT AGENCY > FEDERAL ASSISTANCE

p.44 L22 – p.48 L24p.46 L11 – p.50 L19
-
For activities of the Federal Emergency Management Agency for Federal assistance through grants, contracts, cooperative agreements, and other activities, $3,758,992,810, $3,758,992,810 (increased by $15,000,000) (increased by $30,000,000), which shall be allocated as follows: (1) $520,000,000 for the State Homeland Security Grant Program under section 2004 of the Homeland Security Act of 2002 (6 U.S.C. 605), of which $90,000,000 shall be for Operation Stonegarden and $15,000,000 shall be for Tribal Homeland Security Grants under section 2005 of the Homeland Security Act of 2002 (6 U.S.C. 606): Provided, That notwithstanding subsection (c)(4) of such section 2004, for fiscal year 2025, the Commonwealth of Puerto Rico shall make available to local and tribal governments amounts provided to the Commonwealth of Puerto Rico under this paragraph in accordance with subsection (c)(1) of such section 2004. (2) $615,000,000 for the Urban Area Security Initiative under section 2003 of the Homeland Security Act of 2002 (6 U.S.C. 604). (3) $305,000,000 (reduced by $305,000,000) (increased by $305,000,000) (increased by $80,000,000) (reduced by $80,000,000) (increased by $30,000,000) for the Nonprofit Security Grant Program under section 2009 of the Homeland Security Act of 2002 (6 U.S.C. 609a), of which $152,500,000 (increased by $15,000,000) is for eligible recipients located in high-risk urban areas that receive funding under section 2003 of such Act and $152,500,000 (increased by $15,000,000) is for eligible recipients that are located outside such areas: Provided, That eligible recipients are those described in section 2009(b) of such Act (6 U.S.C. 609a(b)) or are an otherwise eligible recipient at risk of a terrorist or other extremist attack. (4) $105,000,000 for Public Transportation Security Assistance, Railroad Security Assistance, and Over-the-Road Bus Security Assistance under sections 1406, 1513, and 1532 of the Implementing Recommendations of the 9/11 Commission Act of 2007 (6 U.S.C. 1135, 1163, and 1182), of which $10,000,000 shall be for Amtrak security and $2,000,000 shall be for Over-the-Road Bus Security: Provided, That such public transportation security assistance shall be provided directly to public transportation agencies. (5) $100,000,000 for Port Security Grants in accordance with section 70107 of title 46, United States Code. (6) $720,000,000, $720,000,000 (increased by $15,000,000), to remain available until September 30, 2026, of which $360,000,000 (increased by $7,500,000) shall be for Assistance to Firefighter Grants and $360,000,000 (increased by $7,500,000) shall be for Staffing for Adequate Fire and Emergency Response Grants under sections 33 and 34 respectively of the Federal Fire Prevention and Control Act of 1974 (15 U.S.C. 2229 and 2229a). (7) $355,000,000 for emergency management performance grants under the National Flood Insurance Act of 1968 (42 U.S.C. 4001 et seq.), the Robert T. Stafford Disaster Relief and Emergency Assistance Act (42 U.S.C. 5121), the Earthquake Hazards Reduction Act of 1977 (42 U.S.C. 7701), section 762 of title 6, United States Code, and Reorganization Plan No. 3 of 1978 (5 U.S.C. App.). (8) $312,750,000 for necessary expenses for Flood Hazard Mapping and Risk Analysis, in addition to and to supplement any other sums appropriated under the National Flood Insurance Fund, and such additional sums as may be provided by States or other political subdivisions for cost-shared mapping activities under section 1360(f)(2) of the National Flood Insurance Act of 1968 (42 U.S.C. 4101(f)(2)), to remain available until expended. (9) $12,000,000 for Regional Catastrophic Preparedness Grants. (10) $130,000,000 for the emergency food and shelter program under title III of the McKinney- Vento Homeless Assistance Act (42 U.S.C. 11331), to remain available until September 30, 2026: Provided, That not to exceed 3.5 percent shall be for total administrative costs. (11) $40,000,000 for the Next Generation Warning System. (12) $221,343,810 for Community Project Funding grants, which shall be for the purposes, and the amounts, specified in the table entitled ‘‘Homeland Security—Community Project Funding’’ in the report accompanying this Act, of which— (A) $81,771,896, in addition to amounts otherwise made available for such purpose, is for emergency operations center grants under section 614 of the Robert T. Stafford Disaster Relief and Emergency Assistance Act (42 U.S.C. 5196c); and (B) $139,571,914, in addition to amounts otherwise made available for such purpose, is for pre-disaster mitigation grants under section 203 of the Robert T. Stafford Disaster Relief and Emergency Assistance Act (42 U.S.C. 5133(e), notwithstanding subsections (f), (g), and (l) of that section (42 U.S.C. 5133(f), (g), (l)). (13) $322,899,000 to sustain current operations for training, exercises, technical assistance, and other programs.
+
For activities of the Federal Emergency Management Agency for Federal assistance through grants, contracts, cooperative agreements, and other activities, $3,758,992,810, $3,758,992,810 (increased by $15,000,000) (increased by $30,000,000), which shall be allocated as follows: (1) $520,000,000 for the State Homeland Security Grant Program under section 2004 of the Homeland Security Act of 2002 (6 U.S.C. 605), of which $90,000,000 shall be for Operation Stonegarden and $15,000,000 shall be for Tribal Homeland Security Grants under section 2005 of the Homeland Security Act of 2002 (6 U.S.C. 606): Provided, That notwithstanding subsection (c)(4) of such section 2004, for fiscal year 2025, the Commonwealth of Puerto Rico shall make available to local and tribal governments amounts provided to the Commonwealth of Puerto Rico under this paragraph in accordance with subsection (c)(1) of such section 2004. (2) $615,000,000 for the Urban Area Security Initiative under section 2003 of the Homeland Security Act of 2002 (6 U.S.C. 604). (3) $305,000,000 (reduced by $305,000,000) (increased by $305,000,000) (increased by $80,000,000) (reduced by $80,000,000) (increased by $30,000,000) for the Nonprofit Security Grant Program under section 2009 of the Homeland Security Act of 2002 (6 U.S.C. 609a), of which $152,500,000 (increased by $15,000,000) is for eligible recipients located in high-risk urban areas that receive funding under section 2003 of such Act and $152,500,000 (increased by $15,000,000) is for eligible recipients that are located outside such areas: Provided, That eligible recipients are those described in section 2009(b) of such Act (6 U.S.C. 609a(b)) or are an otherwise eligible recipient at risk of a terrorist or other extremist attack. (4) $105,000,000 for Public Transportation Security Assistance, Railroad Security Assistance, and Over-the-Road Bus Security Assistance under sections 1406, 1513, and 1532 of the Implementing Recommendations of the 9/11 Commission Act of 2007 (6 U.S.C. 1135, 1163, and 1182), of which $10,000,000 shall be for Amtrak security and $2,000,000 shall be for Over-the-Road Bus Security: Provided, That such public transportation security assistance shall be provided directly to public transportation agencies. (5) $100,000,000 for Port Security Grants in accordance with section 70107 of title 46, United States Code. (6) $720,000,000, $720,000,000 (increased by $15,000,000), to remain available until September 30, 2026, of which $360,000,000 (increased by $7,500,000) shall be for Assistance to Firefighter Grants and $360,000,000 (increased by $7,500,000) shall be for Staffing for Adequate Fire and Emergency Response Grants under sections 33 and 34 respectively of the Federal Fire Prevention and Control Act of 1974 (15 U.S.C. 2229 and 2229a). (7) $355,000,000 for emergency management performance grants under the National Flood Insurance Act of 1968 (42 U.S.C. 4001 et seq.), the Robert T. Stafford Disaster Relief and Emergency Assistance Act (42 U.S.C. 5121), the Earthquake Hazards Reduction Act of 1977 (42 U.S.C. 7701), section 762 of title 6, United States Code, and Reorganization Plan No. 3 of 1978 (5 U.S.C. App.). (8) $312,750,000 for necessary expenses for Flood Hazard Mapping and Risk Analysis, in addition to and to supplement any other sums appropriated under the National Flood Insurance Fund, and such additional sums as may be provided by States or other political subdivisions for cost-shared mapping activities under section 1360(f)(2) of the National Flood Insurance Act of 1968 (42 U.S.C. 4101(f)(2)), to remain available until expended. (9) $12,000,000 for Regional Catastrophic Preparedness Grants. (10) $130,000,000 for the emergency food and shelter program under title III of the McKinney-Vento Homeless Assistance Act (42 U.S.C. 11331), to remain available until September 30, 2026: Provided, That not to exceed 3.5 percent shall be for total administrative costs. (11) $40,000,000 for the Next Generation Warning System. (12) $221,343,810 for Community Project Funding grants, which shall be for the purposes, and the amounts, specified in the table entitled ‘‘Homeland Security—Community Project Funding’’ in the report accompanying this Act, of which— (A) $81,771,896, in addition to amounts otherwise made available for such purpose, is for emergency operations center grants under section 614 of the Robert T. Stafford Disaster Relief and Emergency Assistance Act (42 U.S.C. 5196c); and (B) $139,571,914, in addition to amounts otherwise made available for such purpose, is for pre-disaster mitigation grants under section 203 of the Robert T. Stafford Disaster Relief and Emergency Assistance Act (42 U.S.C. 5133(e), notwithstanding subsections (f), (g), and (l) of that section (42 U.S.C. 5133(f), (g), (l)). (13) $322,899,000 to sustain current operations for training, exercises, technical assistance, and other programs.
@@ -809,8 +809,7 @@

TITLE V > GENERAL PROVISIONS > SEC. 562

(2) Executive Order 14008, relating to Tackling Climate Crisis at Home and Abroad. (3) Section 6 of Executive Order 14013, relating to Rebuilding and Enhancing Programs To Resettle Refugees and Planning for the Impact of Climate Change on Migration. -(4) Executive Order 14030, relating to Climate- -Related Financial Risk. +(4) Executive Order 14030, relating to Climate-Related Financial Risk. (5) Executive Order 14057, relating to Catalyzing Clean Energy Industries and Jobs Through Federal Sustainability. (6) Executive Order 14082, relating to Implement of the Energy and Infrastructure Provisions of @@ -936,7 +935,7 @@

TITLE V > GENERAL PROVISIONS > SEC. 567

- + + + + + + ', html, re.DOTALL) assert m, "embed missing" data = json.loads(m.group(1).replace("<\\/", " tuple[str, str]: - """(printed display text, merged whole-word text) from the real parser. +def _find_fixture_texts() -> tuple[str, str, dict]: + """(printed display text, merged whole-word text, that page's join points). - Both come from the producer the browser has to agree with — `pdf_full_text` - is the de-hyphenated ground truth the flattened search string must - reproduce, so the fixture can't encode a belief about GPO's line-joining - that the parser doesn't share. + All three come from the producer the browser has to agree with — `pdf_full_text` + is the reflowed ground truth the flattened search string must reproduce, so the + fixture can't encode a belief about GPO's line-joining that the parser doesn't + share. Since #650 the join points travel with the text for the same reason: the + browser applies the producer's decision rather than re-deriving one, so a fixture + that withheld them would be testing a document the pipeline never emits. + + The page is merged with its OWN evidence, exactly as `extract_clean_pages` does. """ from deltatrack.parsers.pdf_text import ( + BreakEvidence, Page, _merge_print_lines, _parse_print_lines, pdf_full_text, pdf_full_text_print, + pdf_print_join_points, ) print_lines = _parse_print_lines(_FIND_PAGE_SRC.rstrip("\n")) - merged, ranges = _merge_print_lines(print_lines) + merged, ranges = _merge_print_lines(print_lines, BreakEvidence.from_print_lines([print_lines])) page = Page(1, tuple(merged), tuple(print_lines), tuple(ranges)) printed_text, _ = pdf_full_text_print([page]) merged_text, _ = pdf_full_text([page]) - return printed_text, merged_text + return printed_text, merged_text, pdf_print_join_points([page]) def _render_find_report() -> str: @@ -1025,7 +1031,7 @@ def _render_find_report() -> str: """ from deltatrack.formatters.diff_html import format_diff_html - printed_text, _ = _find_fixture_texts() + printed_text, _, join_points = _find_fixture_texts() start = printed_text.index("vehicles") canonical = { "schema_version": "2.0", @@ -1036,6 +1042,7 @@ def _render_find_report() -> str: }, "summary": {"added": 0, "removed": 0, "modified": 1}, "full_text": {"v1": "", "v2": printed_text}, + "join_points": {"v1": {"at": [], "drop": ""}, "v2": join_points}, "changes": [ { "id": "c0", @@ -1163,7 +1170,7 @@ def test_find_agrees_with_the_parser_merged_text(chromium, tmp_path): """ import re as _re - _, merged_text = _find_fixture_texts() + _, merged_text, _joins = _find_fixture_texts() # Windows stay inside one merged line. Each merged line is already whole-word # (the parser rejoined its soft hyphens), so this pins the de-hyphenation # contract without asserting how the JS joins one display line to the next. diff --git a/tests/test_pdf_word_break_recall.py b/tests/test_pdf_word_break_recall.py index 193c1936..60ec322b 100644 --- a/tests/test_pdf_word_break_recall.py +++ b/tests/test_pdf_word_break_recall.py @@ -61,12 +61,16 @@ from lxml import etree from pdf_corpus import cached_pages, dual_format_versions -from deltatrack.parsers.pdf_text import Page +from deltatrack.parsers.pdf_text import Page, pdf_full_text, pdf_full_text_print, pdf_print_join_points pytestmark = pytest.mark.slow _RESIDUALS_PATH = Path(__file__).parent / "data" / "pdf" / "word_break_residuals.json" +#: Width of the line-number gutter `pdf_full_text_print` renders before each row +#: (`{number:>5}` plus two spaces), which a reflowing consumer skips past. +_GUTTER_WIDTH = 7 + #: A word token as this check counts one: starts alphanumeric, may carry internal #: hyphens, apostrophes and periods (``E-Verify``, ``U.S.C.``, ``Nation's``). _WORD = re.compile(r"[A-Za-z0-9][A-Za-z0-9'’\-\.]*") @@ -258,3 +262,51 @@ def test_printed_word_breaks_reflow_to_real_words(bill: str, xml_path: Path, pdf ) assert not problems, f"{version}: " + " | ".join(problems) + + +def test_join_points_reproduce_the_reflowed_text() -> None: + """The carried join points are sufficient: applying them reflows the printed text + into exactly the whole-word text, with no re-derivation. + + This is the property #653 asks the document to have — "a consumer may derive by + applying facts the document carries" — and it is the reason the field exists. If + the offsets, their delta encoding, or the drop bits were wrong in any way, the + reconstruction would diverge from `pdf_full_text`, which is computed independently + of them. + + Whitespace-insensitive because the two renderings gutter differently: the printed + one keeps every source line's own margin number, the reflowed one keeps the merged + line's. The characters that matter — which words exist, and which hyphens are in + them — are what this compares. + """ + pages = cached_pages(_CASES[0][2]) + printed, _ = pdf_full_text_print(pages) + reflowed_expected, _ = pdf_full_text(pages) + points = pdf_print_join_points(pages) + + assert points["at"], "no join points emitted for a bill whose printer breaks words" + assert len(points["at"]) == len(points["drop"]) + + offsets: list[int] = [] + running = 0 + for delta in points["at"]: + running += delta + offsets.append(running) + + assert all(printed[o] == "-" for o in offsets), "a join point does not address a hyphen" + + out: list[str] = [] + prev = 0 + for offset, bit in zip(offsets, points["drop"]): + out.append(printed[prev:offset]) + if bit == "0": + out.append("-") # the word's own hyphen survives the join + # Skip to the continuation: past this row's newline, past the blank line that + # separates pages when the break sits at a page seam, then past the gutter. + prev = printed.index("\n", offset) + 1 + if printed[prev : prev + 1] == "\n": + prev += 1 + prev += _GUTTER_WIDTH + out.append(printed[prev:]) + + assert "".join("".join(out).split()) == "".join(reflowed_expected.split()) diff --git a/web/webapp/sample/example.html b/web/webapp/sample/example.html index d7dc90df..391f1740 100644 --- a/web/webapp/sample/example.html +++ b/web/webapp/sample/example.html @@ -926,7 +926,7 @@

TITLE V > GENERAL PROVISIONS > SEC. 567

Homeland Security for the fiscal year ending - - + + + + + ', html, re.DOTALL) assert m, "embed missing" data = json.loads(m.group(1).replace("<\\/", " tuple[str, str, dict]: - """(printed display text, merged whole-word text, that page's join points). +def _find_fixture_texts() -> tuple[str, str]: + """(printed display text, merged whole-word text) from the real parser. - All three come from the producer the browser has to agree with — `pdf_full_text` - is the reflowed ground truth the flattened search string must reproduce, so the - fixture can't encode a belief about GPO's line-joining that the parser doesn't - share. Since #650 the join points travel with the text for the same reason: the - browser applies the producer's decision rather than re-deriving one, so a fixture - that withheld them would be testing a document the pipeline never emits. - - The page is merged with its OWN evidence, exactly as `extract_clean_pages` does. + Both come from the producer the browser has to agree with — `pdf_full_text` + is the de-hyphenated ground truth the flattened search string must + reproduce, so the fixture can't encode a belief about GPO's line-joining + that the parser doesn't share. """ from deltatrack.parsers.pdf_text import ( - BreakEvidence, Page, _merge_print_lines, _parse_print_lines, pdf_full_text, pdf_full_text_print, - pdf_print_join_points, ) print_lines = _parse_print_lines(_FIND_PAGE_SRC.rstrip("\n")) - merged, ranges = _merge_print_lines(print_lines, BreakEvidence.from_print_lines([print_lines])) + merged, ranges = _merge_print_lines(print_lines) page = Page(1, tuple(merged), tuple(print_lines), tuple(ranges)) printed_text, _ = pdf_full_text_print([page]) merged_text, _ = pdf_full_text([page]) - return printed_text, merged_text, pdf_print_join_points([page]) + return printed_text, merged_text def _render_find_report() -> str: @@ -1031,7 +1025,7 @@ def _render_find_report() -> str: """ from deltatrack.formatters.diff_html import format_diff_html - printed_text, _, join_points = _find_fixture_texts() + printed_text, _ = _find_fixture_texts() start = printed_text.index("vehicles") canonical = { "schema_version": "2.0", @@ -1042,7 +1036,6 @@ def _render_find_report() -> str: }, "summary": {"added": 0, "removed": 0, "modified": 1}, "full_text": {"v1": "", "v2": printed_text}, - "join_points": {"v1": {"at": [], "drop": ""}, "v2": join_points}, "changes": [ { "id": "c0", @@ -1170,7 +1163,7 @@ def test_find_agrees_with_the_parser_merged_text(chromium, tmp_path): """ import re as _re - _, merged_text, _joins = _find_fixture_texts() + _, merged_text = _find_fixture_texts() # Windows stay inside one merged line. Each merged line is already whole-word # (the parser rejoined its soft hyphens), so this pins the de-hyphenation # contract without asserting how the JS joins one display line to the next. diff --git a/tests/test_pdf_word_break_recall.py b/tests/test_pdf_word_break_recall.py index 81f0cc97..c1c1cae0 100644 --- a/tests/test_pdf_word_break_recall.py +++ b/tests/test_pdf_word_break_recall.py @@ -61,16 +61,12 @@ from lxml import etree from pdf_corpus import cached_pages, dual_format_versions -from deltatrack.parsers.pdf_text import Page, pdf_full_text, pdf_full_text_print, pdf_print_join_points +from deltatrack.parsers.pdf_text import Page pytestmark = pytest.mark.slow _RESIDUALS_PATH = Path(__file__).parent / "data" / "pdf" / "word_break_residuals.json" -#: Width of the line-number gutter `pdf_full_text_print` renders before each row -#: (`{number:>5}` plus two spaces), which a reflowing consumer skips past. -_GUTTER_WIDTH = 7 - #: A word token as this check counts one: starts alphanumeric, may carry internal #: hyphens, apostrophes and periods (``E-Verify``, ``U.S.C.``, ``Nation's``). _WORD = re.compile(r"[A-Za-z0-9][A-Za-z0-9'’\-\.]*") @@ -358,53 +354,6 @@ def test_printed_word_breaks_reflow_to_real_words(bill: str, xml_path: Path, pdf assert not problems, f"{version}: " + " | ".join(problems) -def test_join_points_reproduce_the_reflowed_text() -> None: - """The carried join points are sufficient: applying them reflows the printed text - into exactly the whole-word text, with no re-derivation. - - This is the property #653 asks the document to have — "a consumer may derive by - applying facts the document carries" — and it is the reason the field exists. If - the offsets, their delta encoding, or the drop bits were wrong in any way, the - reconstruction would diverge from `pdf_full_text`, which is computed independently - of them. - - Whitespace-insensitive because the two renderings gutter differently: the printed - one keeps every source line's own margin number, the reflowed one keeps the merged - line's. The characters that matter — which words exist, and which hyphens are in - them — are what this compares. - """ - pages = cached_pages(_CASES[0][2]) - printed, _ = pdf_full_text_print(pages) - reflowed_expected, _ = pdf_full_text(pages) - points = pdf_print_join_points(pages) - - assert points["at"], "no join points emitted for a bill whose printer breaks words" - assert len(points["at"]) == len(points["drop"]) - - offsets: list[int] = [] - running = 0 - for delta in points["at"]: - running += delta - offsets.append(running) - - assert all(printed[o] == "-" for o in offsets), "a join point does not address a hyphen" - - out: list[str] = [] - prev = 0 - for offset, bit in zip(offsets, points["drop"]): - out.append(printed[prev:offset]) - if bit == "0": - out.append("-") # the word's own hyphen survives the join - # Skip to the continuation: past this row's newline, past the blank line that - # separates pages when the break sits at a page seam, then past the gutter. - prev = printed.index("\n", offset) + 1 - if printed[prev : prev + 1] == "\n": - prev += 1 - prev += _GUTTER_WIDTH - out.append(printed[prev:]) - - assert "".join("".join(out).split()) == "".join(reflowed_expected.split()) - # --- Negative controls ----------------------------------------------------------- # Three ways a repair can look correct while being wrong. Each pins one, and each was diff --git a/web/webapp/sample/example.html b/web/webapp/sample/example.html index 391f1740..d7dc90df 100644 --- a/web/webapp/sample/example.html +++ b/web/webapp/sample/example.html @@ -926,7 +926,7 @@

TITLE V > GENERAL PROVISIONS > SEC. 567

Homeland Security for the fiscal year ending - - +