feat(docx-markdoc): verify create output with an independent round-trip oracle by stevenobiajulu · Pull Request #1189 · UseJunior/safe-docx

stevenobiajulu · 2026-10-08T22:41:24Z

Problem (#1185)

create's read-back and footer checks compare the DOCX with the plain-text projection of the lowered DocumentSpec. A lowering bug that drops a word, paragraph, table cell or footer from both the spec and its projection passes them. The optional PDF check only looks for the first and last paragraphs and the footers, so it can't see a dropped middle word either.

Change

Independent round trip (create/oracle.ts, certificate checks.roundTrip).
- The expected text comes only from the original Markdoc, through the oracle's own AST walk. It never touches the lowering, theme, engine or DocumentSpec.
- The actual text comes from re-importing the DOCX through docx-markdoc import.
- Body paragraphs and each section's footer paragraphs are compared as sequences using a Myers alignment.
- Normalization is Unicode NFC and whitespace collapsing only: no case, punctuation or duplicate folding.
- Mismatches are recorded as missing, extra or changed, each with its source line. A changed mismatch also records the missing and extra word spans, with four words of context on each side.
- Negative controls: a deleted word, paragraph, table cell and footer text must each be detected.
- The existing read-back, footer and brownfield checks are kept.
PDF word check (pdf.words).
- When a PDF renders, its text layer (pdftotext -raw) is aligned word by word with the source.
- Generated text is modelled from the source, not excused by its shape:
  - List labels (1., (a), (i), bullets) are expected words, computed from the grammar's numbering.
  - Footers: each page must begin with exactly its section's effective footer text and its own page number (pages count from 1). This is checked after alignment against the section of the content on that page.
  - Table headers: a table's header row may repeat only at the top of a page where that same table continues on both sides of it.
- Failures: any other missing or extra word, or a page with a missing, wrong-section or misnumbered footer, fails with CREATION_PDF_WORDS_MISMATCH. Texts too different to align fail as over-budget.
- Disclosure: the certificate's limitations string names the remaining ambiguity (page-to-section assignment). It can only cause false failures, never false passes.
Failure report.
- When a check finds a mismatch, the run publishes nothing and leaves existing outputs byte-identical, including an earlier <stem>.verification.json.
- It writes <stem>.failed-verification.json and names it in the error.
- The report is staged and renamed under the stem lock, so it is never written through a symlink.
- The next failed run replaces the report; the next successful run removes it.
The README, design, proposal and tasks are updated, and there are three new scenarios (SDX-MDOC-CREATE-11/12/13).

Close calls (raised with Lantern 3 before merge)

New public file convention: <stem>.failed-verification.json, as described above. Missing PDF tools under --require-pdf and validation errors write no report: they are not mismatches, and validation errors already carry line numbers.
The PDF word check runs whenever a PDF renders, not only under --require-pdf. A rendered PDF that fails its existing check already fails the build in default mode, and publishing a PDF that dropped content would be worse.
renderPlainPdf now runs pdftotext -raw.
- -raw reads content-stream order, which for LibreOffice output is document order.
- The default reading-order mode, verified against real LibreOffice output, reads a 2×2 table column by column, and scatters a justified line that ends in a manual break past the footer.
- Required-text needles are unaffected.
- PlainPdfVerdict gains pageTexts. The certificate omits it.

Verification

Tests: docx-markdoc 265/265, plus spec coverage, allure labels, eslint and strict OpenSpec.
Review round 1 (.peer-review/greenfield-pr1189-20261008-164211-review.md): two P1s and two P2s, all fixed in 8a31bb4 with regression tests:
- P1: a generated list number could stand in for a deleted literal 1.;
- P1: a numeric footer was read as the page number;
- P2: a repeated table header false-failed a valid document;
- P2: a footer lost on a later page passed.
Red checks: each probe fails without the piece it tests:
- repeated-header acceptance;
- footer-region validation;
- list labels as expected words;
- header position validation;
- an oracle fed from the lowering;
- default (non--raw) pdftotext.
Real LibreOffice probes:
- every construct;
- wrapped table cells;
- a 65-row table across two pages (header repeated);
- inherited, unlinked and empty section footers with page numbers;
- a numeric footer 123;
- the benchmark resolution (201 expected words).

Refs #1185
Refs #1162

vercel · 2026-10-08T22:41:31Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
site	Ready	Preview	Oct 9, 2026 2:17am UTC

usejunior-llm-gate · 2026-10-08T22:43:11Z

LLM gate (advisory)

All evaluated rules passed - 5 pass, 0 warn, 0 error, 11 skipped, 16 total

Findings

None.

All 16 rules (5 evaluated, 11 skipped)

Rule	Verdict	Detail
`read_file` response metadata parity	SKIPPED	paths not touched by this PR
Live DOM namespace-safe OOXML writes	SKIPPED	paths not touched by this PR
Complex-field revisions preserve complete accept/reject state machines	PASS	The PR does not touch field atomization, `validateFieldStructure`, `w:fldChar`, `w:instrText`, `w:delInstrText`, or collapsed-field comparison logic, as its changes are strictly focused on Markdoc document creation round-trip verification.
Field validation per story, not global	SKIPPED	paths not touched by this PR
Revision IDs seeded from all revision-bearing side parts	SKIPPED	paths not touched by this PR
Accept/reject sweep side parts and caches	PASS	The PR only modifies Markdoc document creation and verification logic under packages/docx-markdoc, and does not touch acceptChanges, rejectChanges, REVISION_STORY_PART_PATHS, or side-part revision markup.
`DocumentViewNode.heading` stays canonical	SKIPPED	paths not touched by this PR
AI-author parity across entry points	SKIPPED	paths not touched by this PR
Property-change wrapper discipline	SKIPPED	paths not touched by this PR
`SUPPORT.md` Table A drift vs. implementation	SKIPPED	paths not touched by this PR
Table A / Table B boundary on side-part revisions	SKIPPED	paths not touched by this PR
Canonical-emission surface completeness	SKIPPED	paths not touched by this PR
Unit-test quality (avoid tautological / change-detector tests)	PASS	In packages/docx-markdoc/src/create/roundtrip-oracle.test.ts: a: ok; b: ok; c: ok; d: n/a. Test assertions are constructed from first principles with precise semantic claims, negative controls are used to confirm oracle sensitivity to corruption, and the only mock isolates the external PDF render binary boundary.
Re-derived facts vs canonical sources	PASS	The PR explicitly justifies the independent Markdoc AST walk and projection in openspec/changes/add-markdoc-document-creation/design.md:195-207 to ensure lowering/compiler bugs are detected independently of the spec, and adds robust pinning tests in packages/docx-markdoc/src/create/roundtrip-oracle.test.ts.
`.openspec` tag ↔ test-assertion drift	PASS	Inspected packages/docx-markdoc/src/create/roundtrip-oracle.test.ts (lines 87-581) and confirmed that the test cases for SDX-MDOC-CREATE-11, 12, and 13 fully and independently exercise all corresponding GIVEN/WHEN/THEN clauses with precise assertions and negative controls, showing no tag-stuffing or drift.
Library stays general (no downstream-domain leakage)	SKIPPED	paths not touched by this PR

… shape Review of #1189 found two false passes and two false/missed failures in the PDF word check. Generated text is now derived from the source: - list labels (1., (a), (i), bullets) are expected words, so a generated label cannot stand in for a deleted literal one; - each page must begin with exactly its section's effective footer text and its own page number, checked against the section of the content aligned on that page: a missing, wrong-section or misnumbered footer fails, and a numeric footer is never taken for the page number; - a table header row is accepted again only at the top of a page where that table continues on both sides of it. The shape-based list-number leniency and edge peeling are gone. Refs #1185

Confirmation review of #1189 found two more false passes and one false failure in the PDF word check: - a header-shaped run at a page head that is not a validated repetition must align with its table's own header row, or it is reported as an unverified table header; generated header text can no longer stand in for a lost header-equal body row; - every table with an identical header is tried as the identity of a continuation header; - a page with no aligned text takes every section a consistent page order allows (sections run in order, each on a new page) and must carry the footer of all of them; a section with no page fails. Refs #1185

Review of #1189 showed that header and footer rules checked separately can be satisfied by generated text in layouts the text cannot disambiguate (a table with empty trailing rows followed by a table with the same header), and can reject valid ones. Each page break is now explained jointly: between two aligned source words, every page that starts there must start at a point in the source (including inside wordless rows), which fixes its section footer and page number, the header LibreOffice repeats when a table continues, and the source words at the top of the page. Sections start on new pages. Two observed LibreOffice behaviours are modelled: a header row stays with the first body row, and a last row can spill its empty remainder onto the next page with the header repeated. The check passes only when every fitting placement explains the same source words; ambiguity fails. Refs #1185

…ip oracle The read-back and footer checks compare the DOCX with the projection of the lowered DocumentSpec, so a lowering bug that drops content from both passes them. create now also: - re-imports the DOCX through docx-markdoc import and compares body and footer paragraph sequences with text read from the original Markdoc alone (its own AST walk; NFC and whitespace collapsing only), with deleted-word, -paragraph, -table-cell and -footer negative controls and word-level missing/extra spans recorded in the certificate; - aligns a rendered PDF's text layer (pdftotext -raw, document order) with the source word for word, allowing only each page's footer and page number at a page edge and list numbers (CREATION_PDF_WORDS_MISMATCH); - leaves <stem>.failed-verification.json, named in the error, when a check finds a mismatch; nothing is published, the previous certificate is kept, and the next successful build removes the report. Refs #1185

…ge footer probes Refs #1185

… shape Review of #1189 found two false passes and two false/missed failures in the PDF word check. Generated text is now derived from the source: - list labels (1., (a), (i), bullets) are expected words, so a generated label cannot stand in for a deleted literal one; - each page must begin with exactly its section's effective footer text and its own page number, checked against the section of the content aligned on that page: a missing, wrong-section or misnumbered footer fails, and a numeric footer is never taken for the page number; - a table header row is accepted again only at the top of a page where that table continues on both sides of it. The shape-based list-number leniency and edge peeling are gone. Refs #1185

Confirmation review of #1189 found two more false passes and one false failure in the PDF word check: - a header-shaped run at a page head that is not a validated repetition must align with its table's own header row, or it is reported as an unverified table header; generated header text can no longer stand in for a lost header-equal body row; - every table with an identical header is tried as the identity of a continuation header; - a page with no aligned text takes every section a consistent page order allows (sections run in order, each on a new page) and must carry the footer of all of them; a section with no page fails. Refs #1185

When one table's header is a prefix of another's, a continuation header removed only the shorter prefix, leaving the rest available to stand in for a lost body word. Each page's candidate now offers every matching header length, longest first, advancing only when one fails its continuation check. A PDF with no pages fails the footer check. Refs #1185

Review of #1189 showed that header and footer rules checked separately can be satisfied by generated text in layouts the text cannot disambiguate (a table with empty trailing rows followed by a table with the same header), and can reject valid ones. Each page break is now explained jointly: between two aligned source words, every page that starts there must start at a point in the source (including inside wordless rows), which fixes its section footer and page number, the header LibreOffice repeats when a table continues, and the source words at the top of the page. Sections start on new pages. Two observed LibreOffice behaviours are modelled: a header row stays with the first body row, and a last row can spill its empty remainder onto the next page with the header repeated. The check passes only when every fitting placement explains the same source words; ambiguity fails. Refs #1185

codecov · 2026-10-09T01:14:46Z

Codecov Report

❌ Patch coverage is 90.63098% with 49 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
packages/docx-markdoc/src/create/oracle.ts	91.49%	6 Missing and 31 partials ⚠️
packages/docx-markdoc/src/create/cli-create.ts	82.50%	1 Missing and 6 partials ⚠️
packages/docx-markdoc/src/create/create.ts	88.09%	1 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

docx-markdoc rose above its floor by more than the allowed drift (lines 92.25 -> 93.32, branches 79.06 -> 80.97) with the round-trip oracle tests; values are taken from the workspace-test (20) coverage artifact for this head. Refs #1185

…mpty pages Final review of #1189: - pdftotext -raw emits a row split across pages as each cell's fragment on each page. Such a row is read back in cell order only when the end of one page and the start of the next (and any whole pages between) partition exactly into the row's cells, every cell starting on the first page; words are only reordered, so a lost or duplicated word still fails. - Every page must hold content (source between its start and the next, a spilled row remainder, or the aligned text that follows), so an empty page cannot be invented to move a section start and its footer. - Running out of search budget is an explicit failure. Refs #1185

…rs; bound the search Final review round 2 of #1189: - The pages holding the rest of a split table row stay pinned to it: each must start inside the row, belong to its section and carry that section's footer, and show the repeated header unless the split row is the header itself. Reading the row back in cell order no longer erases which pages held it. - Header rows that split across pages are joined like body rows. - The fragment search tries a candidate only when it holds exactly the row's words, is memoized and pruned, and runs under a shared budget whose exhaustion fails the check. Refs #1185

Final review round 3 of #1189: a pinned continuation page at a section boundary could stand in for the next section's first page, and text from that next section on the same physical page was not checked against the row's section. Pinned pages now never satisfy a section start, and any other text aligned on a pinned page must belong to the row's section. Refs #1185

Releases the docx-markdoc create follow-ups: explicit fill-in markup with literal bare brackets (#1187, #1184), the independent round-trip oracle, word-level PDF check and failure report (#1189, #1185), and the edited-DOCX guard for --replace (#1190, #1186), plus the header/footer carry-forward fix (#1188). Versions bumped with scripts/bump_version.mjs; the CHANGELOG entry is hand-written. Refs #1162

vercel Bot deployed to Preview October 8, 2026 22:44

View deployment

vercel Bot deployed to Preview October 8, 2026 22:54

View deployment

vercel Bot deployed to Preview October 8, 2026 23:10

View deployment

vercel Bot deployed to Preview October 9, 2026 00:14

View deployment

vercel Bot deployed to Preview October 9, 2026 00:37

View deployment

Base automatically changed from 1184-explicit-fill-ins-20261008 to main October 9, 2026 00:55

github-actions Bot added the feat PR type: feature label Oct 9, 2026

stevenobiajulu added 6 commits October 8, 2026 18:55

test(docx-markdoc): pin report replacement under the lock and page-ed…

54fb753

…ge footer probes Refs #1185

stevenobiajulu force-pushed the 1185-independent-roundtrip-oracle-20261008 branch from e0d2cae to 4063e0a Compare October 9, 2026 00:55

vercel Bot deployed to Preview October 9, 2026 00:56

View deployment

vercel Bot deployed to Preview October 9, 2026 01:15

View deployment

vercel Bot deployed to Preview October 9, 2026 01:43

View deployment

vercel Bot deployed to Preview October 9, 2026 02:05

View deployment

vercel Bot deployed to Preview October 9, 2026 02:17

View deployment

stevenobiajulu merged commit 8c0f72d into main Oct 9, 2026
27 checks passed

stevenobiajulu deleted the 1185-independent-roundtrip-oracle-20261008 branch October 9, 2026 02:34

This was referenced Oct 9, 2026

docx-markdoc create: independent round-trip oracle from the original Markdoc, plus a word-level PDF check #1185

Closed

chore(release): bump workspace versions to 0.25.0 #1191

Merged

Navigation Menu

Repository navigation

feat(docx-markdoc): verify create output with an independent round-trip oracle - #1189

feat(docx-markdoc): verify create output with an independent round-trip oracle#1189
stevenobiajulu merged 10 commits into
mainfrom
1185-independent-roundtrip-oracle-20261008

stevenobiajulu commented Oct 8, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

usejunior-llm-gate Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

stevenobiajulu commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem (#1185)

Change

Close calls (raised with Lantern 3 before merge)

Verification

Uh oh!

vercel Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

usejunior-llm-gate Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LLM gate (advisory)

Findings

Uh oh!

codecov Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

stevenobiajulu commented Oct 8, 2026 •

edited

Loading

vercel Bot commented Oct 8, 2026 •

edited

Loading

usejunior-llm-gate Bot commented Oct 8, 2026 •

edited

Loading

codecov Bot commented Oct 9, 2026 •

edited

Loading