Skip to content

fix(97): guard degenerate table-row repetition at the agentic pre-flush seam - #99

Merged
r-uben merged 1 commit into
mainfrom
fix/97-table-row-repetition-guard
Jul 30, 2026
Merged

fix(97): guard degenerate table-row repetition at the agentic pre-flush seam#99
r-uben merged 1 commit into
mainfrom
fix/97-table-row-repetition-guard

Conversation

@r-uben

@r-uben r-uben commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

Guards degenerate table-row repetition at the agentic pre-flush seam.

Page 62 of the OBR EFO November 2022 reference run emitted 865 copies of the
15-character row | | | | | | | |, which reached 867 of the 3177 lines of the
assembled markdown (27% of the document)
with no audit event and no doc-level
signal. Page 48 shipped a wholly contentless 12-column table the same way.

Root cause — not a missing guard, a broken one

A repetition guard already existed. It had two independent defects:

  1. OutputNormalizer._RE_LINE_REPEAT is ^(.{20,})\n(?:\1\n){4,} — the
    20-character floor means a narrow table's empty row (15 chars) can never
    match. Undocumented, and untested, which is how it survived.
  2. normalize() runs at the engine boundary (engines/base.py:587), before
    dual-pass, header repair and table reconstruction. Degenerate rows those stages
    introduce or preserve are never re-checked — which is why page 48's
    37-character row, long enough to match the regex, also survived.

Implementation

  • collapse_repeated_table_rows() in core/normalizer.py — table rows only, no
    width floor, pure and unit-testable.
  • _guard_agentic_page_table_repetition called from the agentic pre-flush seam
    beside the existing Figures pipeline broken: placeholder image refs + figures extracted for only some docs #86 _sanitize_agentic_page_image_refs, so the fragment,
    the sidecar and the stitched body all see one text. Byte-identity with
    whole-doc assembly is preserved
    bo is the same object held in
    state.pages[n].best_output, so _rewrite_all_fragments reads the mutated text.
  • table_row_repetition_truncated audit event per page, plus
    _repetition_truncated_note naming affected pages in the document-level error
    surface.

_RE_LINE_REPEAT is deliberately left alone rather than widened: it is on
every engine's hot path and has no tests. Tracked in #98.

The bound is derived, not guessed

Measured across the 15 table pages of the reference document: 13 have a longest
consecutive-identical-row run of exactly 1
, and the only exceptions are the two
defects above (15 and 864). Legitimate tables in this corpus never repeat a row.
MAX_CONSECUTIVE_IDENTICAL_TABLE_ROWS = 2 keeps one copy of slack for genuine
spacer rows in other layouts.

Test plan — observed results

  • pytest tests/test_gh97_table_row_repetition.py -q14 passed
  • pytest tests/ -q1249 passed, 1 xfailed
  • ruff format --check + ruff check on all three touched files → clean
  • Against the real fragments:
    • page 62: 905 → 43 lines, 862 removed, 0 distinct lines lost
    • page 48: 147 → 134 lines, 13 removed, 0 distinct lines lost

Tests call the two static methods directly rather than driving _phase_agentic,
so no provider ladder is needed and they cannot depend on ollama being installed
(the CI trap named in CLAUDE.md).

Reviewer notes

Closes #97

…sh seam

Page 62 of the OBR EFO November 2022 reference run emitted 865 copies of the
15-character row `| | | | | | | |`, which reached 867 of the 3177 lines of the
assembled markdown (27% of the document) with no audit event and no doc-level
signal. Page 48 shipped a wholly contentless 12-column table the same way.

Root cause is two independent defects in the one guard that already existed:

1. `OutputNormalizer._RE_LINE_REPEAT` only matches lines of 20+ characters, so a
   narrow table's empty row can never match it. Untested, which is how the floor
   survived.
2. `normalize()` runs at the engine boundary (`engines/base.py`), before
   dual-pass, header repair and table reconstruction. Degenerate rows those
   stages introduce or preserve are never re-checked.

Adds `collapse_repeated_table_rows` (table rows only, no width floor) and calls
it from a new pre-flush seam beside the GH-86 image-ref sanitizer, so the
fragment, the sidecar and the stitched body all see one text and byte-identity
with whole-doc assembly is preserved. Emits a `table_row_repetition_truncated`
audit event per page and names affected pages in the document-level error
surface, so a consumer gating on metadata.json sees it without parsing the
audit log.

The bound is derived, not guessed: across the 15 table pages of the reference
document, 13 have a longest consecutive-identical-row run of exactly 1, and the
only exceptions are the two defects above (15 and 864). Two copies are kept as
slack for genuine spacer rows in other layouts.

Removal is content-safe by construction - every dropped line is byte-identical
to one that is kept. Verified on the real fragments: page 62 905 -> 43 lines,
page 48 147 -> 134, zero distinct lines lost in either.

Tests: 14 new (pure function, orchestrator seam, doc-level note), called
directly rather than through _phase_agentic so no provider ladder is needed in
CI. Full suite 1249 passed, 1 xfailed. ruff format + check clean.

Closes #97
@r-uben
r-uben force-pushed the fix/97-table-row-repetition-guard branch from 129c5cd to 912bdf2 Compare July 30, 2026 14:21
@r-uben
r-uben merged commit 33fda23 into main Jul 30, 2026
2 checks passed
@r-uben
r-uben deleted the fix/97-table-row-repetition-guard branch July 30, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(tables): degenerate row repetition ships 865 blank rows into the assembled .md (27% of document)

1 participant