fix(97): guard degenerate table-row repetition at the agentic pre-flush seam - #99
Merged
Merged
Conversation
This was referenced Jul 30, 2026
Merged
…sh seam Page 62 of the OBR EFO November 2022 reference run emitted 865 copies of the 15-character row `| | | | | | | |`, which reached 867 of the 3177 lines of the assembled markdown (27% of the document) with no audit event and no doc-level signal. Page 48 shipped a wholly contentless 12-column table the same way. Root cause is two independent defects in the one guard that already existed: 1. `OutputNormalizer._RE_LINE_REPEAT` only matches lines of 20+ characters, so a narrow table's empty row can never match it. Untested, which is how the floor survived. 2. `normalize()` runs at the engine boundary (`engines/base.py`), before dual-pass, header repair and table reconstruction. Degenerate rows those stages introduce or preserve are never re-checked. Adds `collapse_repeated_table_rows` (table rows only, no width floor) and calls it from a new pre-flush seam beside the GH-86 image-ref sanitizer, so the fragment, the sidecar and the stitched body all see one text and byte-identity with whole-doc assembly is preserved. Emits a `table_row_repetition_truncated` audit event per page and names affected pages in the document-level error surface, so a consumer gating on metadata.json sees it without parsing the audit log. The bound is derived, not guessed: across the 15 table pages of the reference document, 13 have a longest consecutive-identical-row run of exactly 1, and the only exceptions are the two defects above (15 and 864). Two copies are kept as slack for genuine spacer rows in other layouts. Removal is content-safe by construction - every dropped line is byte-identical to one that is kept. Verified on the real fragments: page 62 905 -> 43 lines, page 48 147 -> 134, zero distinct lines lost in either. Tests: 14 new (pure function, orchestrator seam, doc-level note), called directly rather than through _phase_agentic so no provider ladder is needed in CI. Full suite 1249 passed, 1 xfailed. ruff format + check clean. Closes #97
r-uben
force-pushed
the
fix/97-table-row-repetition-guard
branch
from
July 30, 2026 14:21
129c5cd to
912bdf2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Guards degenerate table-row repetition at the agentic pre-flush seam.
Page 62 of the OBR EFO November 2022 reference run emitted 865 copies of the
15-character row
| | | | | | | |, which reached 867 of the 3177 lines of theassembled markdown (27% of the document) with no audit event and no doc-level
signal. Page 48 shipped a wholly contentless 12-column table the same way.
Root cause — not a missing guard, a broken one
A repetition guard already existed. It had two independent defects:
OutputNormalizer._RE_LINE_REPEATis^(.{20,})\n(?:\1\n){4,}— the20-character floor means a narrow table's empty row (15 chars) can never
match. Undocumented, and untested, which is how it survived.
normalize()runs at the engine boundary (engines/base.py:587), beforedual-pass, header repair and table reconstruction. Degenerate rows those stages
introduce or preserve are never re-checked — which is why page 48's
37-character row, long enough to match the regex, also survived.
Implementation
collapse_repeated_table_rows()incore/normalizer.py— table rows only, nowidth floor, pure and unit-testable.
_guard_agentic_page_table_repetitioncalled from the agentic pre-flush seambeside the existing Figures pipeline broken: placeholder image refs + figures extracted for only some docs #86
_sanitize_agentic_page_image_refs, so the fragment,the sidecar and the stitched body all see one text. Byte-identity with
whole-doc assembly is preserved —
bois the same object held instate.pages[n].best_output, so_rewrite_all_fragmentsreads the mutated text.table_row_repetition_truncatedaudit event per page, plus_repetition_truncated_notenaming affected pages in the document-level errorsurface.
_RE_LINE_REPEATis deliberately left alone rather than widened: it is onevery engine's hot path and has no tests. Tracked in #98.
The bound is derived, not guessed
Measured across the 15 table pages of the reference document: 13 have a longest
consecutive-identical-row run of exactly 1, and the only exceptions are the two
defects above (15 and 864). Legitimate tables in this corpus never repeat a row.
MAX_CONSECUTIVE_IDENTICAL_TABLE_ROWS = 2keeps one copy of slack for genuinespacer rows in other layouts.
Test plan — observed results
pytest tests/test_gh97_table_row_repetition.py -q→ 14 passedpytest tests/ -q→ 1249 passed, 1 xfailedruff format --check+ruff checkon all three touched files → cleanTests call the two static methods directly rather than driving
_phase_agentic,so no provider ladder is needed and they cannot depend on ollama being installed
(the CI trap named in CLAUDE.md).
Reviewer notes
byte-identical to one that is kept. Note the invariant is "no distinct line
lost", which is weaker than "no information lost": row multiplicity can
change, and
test_no_content_can_be_lostuses set equality so it would notcatch that. On a native born-digital page with genuinely repeated unlabelled
rows this is real content loss. Tracked in Follow-up to #97: repetition guard is agentic-only and unscoped w.r.t. native pages (row multiplicity) #98.
pipeline/orchestrator.py— check the seam placement againstthe byte-identity golden tests.
Closes #97