Conversation
compare() refused any pair whose content controls differed only by w:sdtPr/w:id, with "comparison cannot revise content-control properties". Word and Google Docs renumber that id when they save, so two versions of a document with a table-of-contents control could not be compared at all, although nothing in the control had changed. The id was part of control_property_signature, the one tuple that drives control alignment, the refusal and the accept and reject postconditions. Leaving it out of that tuple keeps the three consistent. Controls that differ only by id now align as unchanged and the redline keeps the original w:sdtPr bytes, and a change inside such a control is revised like any other. A w:sdtPr that holds none of the compared properties reads like no w:sdtPr, so a control that gains or loses an id-only w:sdtPr compares too. A differing w:tag, alias, control type or data binding still refuses the pair. HLD 03 now says the id is not part of the control shell. GitHub issue tensorbee#159.
A run rewritten by try_replace_text lost xml:space="preserve" on its w:t, because replace_in_single_run and replace_across_runs recomputed the flag from the new text alone. Google Docs writes the flag on every w:t, so a no-op replacement, common in a scripted pass that normalizes wording, turned an unchanged run into a deletion and an insertion once the edited copy was compared against its source. The three rewrite sites, which regex replacement shares, now keep a flag the producer wrote and still add one when the new text starts or ends with a space. compare() also read the flag as content wherever it appeared, because the run signature formatted CT_Text whole. The flag only changes how text reads when whitespace sits at an edge, so the text signature now counts it only there. The same signature serves alignment and the accept and reject postconditions, so they stay consistent, and a run that matches keeps the original bytes. Word and character granularity and the ignore options compare text units, which the redline writes back as their own w:t. Each unit copied the flag of its text, so a space split out of a flagged text differed from the same space in an unflagged one. A unit with whitespace at an edge now always carries the flag, so on that path whitespace reads as whitespace whatever the source flag was, and the flag is never content. Files from two producers that place the flag differently then compare the same way at every granularity, and a space split out of a text keeps the flag in the redline, where Word used to drop it. On that path, the shortcut that keeps runs whole when every run of a paragraph matches reads the flag the same way. Otherwise a run that differed only by the flag at an edge was rewritten one unit per run, which moved a bookmark end indexed by run and refused the pair. GitHub issue tensorbee#160.
Comparing a file whose w:pgSz carries w:orient="portrait" against its rdocx-edited copy reported a section_property_change next to the real edit. The pgSz writer emits w:orient only for landscape, so a modelled edit drops the default value, and section_properties_xml compared the two sections as whole CT_SectPr values, Some(Portrait) against None. Portrait is the schema default, so both modelled sections now read it as absent before the equality test. A paragraph that holds a bookmark, hyperlink, comment range or control compares its whole paragraph properties, section break included, so the same rule applies there. Without it such a section-break paragraph kept a formatting diagnostic, or refused the pair when its text changed. A real change to landscape is still a section property revision. The serializer is left alone, because the typed defaults of every generated document carry Some(Portrait) and writing it would move the document.xml hash baselines. GitHub issue tensorbee#160.
compare() failed with "complex field source has no end boundary" on a file against its own copy once update_page_fields had refreshed a field packed in one run, as Google Docs writes footer fields. The refreshed result lands inside that run and changes w:dirty, so compare_field_xml extracts the result on both sides, and complex_field_result looked for the end character only after the end of the run holding separate. In a packed run the end sits inside that same run. complex_field_result now splits the run so that separate ends a run and end starts one before it reads the result. The new runs repeat the original start tag and run properties, and a run with nothing on the far side of the character is left alone, so a field whose separate and end each have a run of their own yields the same bytes as before. The postconditions read a field as its owner only, so the split does not reach them. The same split fixes a quieter loss. update_page_fields writes the result of an uncached one-run-per-part field into the run of end, and the redline then carried an empty insertion without the new page number. It now inserts the result. GitHub issue tensorbee#160.
The comparison fixes, the kept xml:space flag and their tests grow the rdocx and rdocx-oxml packages, so the crates.io archive rows in README.md and crates/rdocx-oxml/README.md and their ARCHIVE_MEASUREMENTS entries are re-measured. GitHub issues tensorbee#159 and tensorbee#160.
This was referenced Sep 27, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
w:idwhen comparing documents. It no longertakes part in
control_property_signature, and aw:sdtPrthat holds noneof the compared properties reads like no
w:sdtPr. A pair whose controlsdiffer only by id, including an id-only
w:sdtPragainst none, compares,and the redline keeps the original
w:sdtPr. HLD 03 now says the id is notpart of the control shell.
xml:space="preserve"on runs rewritten bytry_replace_textandregex replacement. In compare, count the flag in the whole-run signature
only when the text has whitespace at an edge, and give every word,
character or ignore-option unit with edge whitespace the flag, so the flag
is never content on that attributed path. The shortcut of that path that
keeps matching runs whole reads the flag the same way.
w:orient="portrait"as absent when comparing sectionproperties, both for a section on its own and for the section break of a
paragraph that takes the complex path (bookmark, hyperlink, comment range
or control). The orientation that an edited save drops is no longer a
section_property_change, a formatting diagnostic or a refusal.separateends a run andendstarts one before compare extracts its result. Such a field is then read
like the same field written one run per part.
rdocxandrdocx-oxml.Part of #159. This fixes the
w:idhalf of section 2. Section 1 (TOCrebuild), the
w:taghalf of section 2, section 3 (table-row identityattributes) and the request to keep the matrix as a test remain.
Part of #160. This fixes section 2 (both cases) and section 4. Section 1
(content controls reached by replace and text), section 3 (empty
comments.xml), the fidelity paragraph at the end of section 2 and therequest to keep the matrix as a test remain.
Why
Each case turned producer serialization into a refused pair or a spurious
revision.
compare_control_from_xmlrefused the whole pair whencontrol_property_signaturediffered, and the signature held the numericw:id, which Word and Google Docs renumber on save. A TOC control thatgained an id between two saves made the two versions impossible to compare.
placeholder.rsrecomputedpreserve_spacefromthe new text only, and
run_content_signatureformattedCT_Textwhole,flag included. A no-op replacement on a Google Docs file then showed an
unchanged run as deleted and inserted.
pgSzwriter emitsw:orientonly for landscape, and compare lookedat whole
CT_SectPrvalues, soSome(Portrait)againstNonebecame asection property revision.
complex_field_resultsearched for theendcharacter only after the runholding
separate. In a packed run, theendsits inside that same run,so compare failed with "complex field source has no end boundary" once
update_page_fieldshad written a result into the run.Notes
alignment, the refusal and the accept and reject postconditions, so removing
the id there keeps all three consistent. A
w:sdtPrwith none of the tupleproperties now maps to
None, which makes an id-onlyw:sdtPrequal to amissing one. Properties outside the tuple, such as a lock, were already
ignored when both sides had a
w:sdtPr. Thew:tagbehaviour is unchangedon purpose. A differing tag, alias, control type or data binding still
refuses the pair. That refusal is a documented contract (HLD 03 on control
shells, pinned by
comparison_revises_nested_control_content_without_replacing_its_shell),and 159-2b needs the maintainer to classify each
w:sdtPrproperty asidentity, metadata that becomes a diagnostic, or structure. That belongs to
a later PR.
while an inserted control keeps its edited-side id, so the redline can hold
two
w:sdtwith the same id (original A with id 5, edited A with id 7 and anew B with id 5). The postconditions ignore ids and Word renumbers
duplicates on open. The pair used to be refused. Renumbering a colliding
inserted control is left out to keep the diff small.
xml:space, whole-run path. The default path counts the flag only when thetext starts or ends with XML whitespace (space, tab, CR or LF), where it
decides whether Word keeps that whitespace. Internal whitespace does not
make it count. The producer side keeps a flag the source had and, as
before, adds one when the new text starts or ends with a space, which
matches
CT_Text::new.xml:space, attributed path (word or character granularity, or any ignoreoption).
granular_textused to copy the text flag onto every unit. Withthe new signature, a whitespace unit split out of a flagged text then
differed from the same unit of an unflagged text, and the granular
postcondition refused pairs that the whole-run fast path had matched (for
example
<w:t xml:space="preserve">two words</w:t>against<w:t>two words</w:t>at word granularity). Each unit now carries the flagwhen it has edge whitespace, as the review suggested. On this path the flag
is therefore never content, and a split-out space is written with the flag
in the redline, where it used to be written as
<w:t> </w:t>and Worddropped it. A position-aware variant (keep the source flag only on the
first and last unit) was tried and rejected: it breaks
granular_hyperlink_edits_preserve_the_owner_shellandinline_control_runs_use_granularity_and_left_biased_ignores, which pin anunflagged
<w:t>alpha </w:t><w:t>beta</w:t>as equal to<w:t>alpha beta</w:t>. One consequence to know about: a pair that differsonly by the flag on edge whitespace, such as
WORDflagged againstunflagged, gives a deletion and an insertion with default options and no
revision on the attributed path.
xml:space, whole-run shortcut of the attributed path (found in review).compare_granular_paragraphkeeps every run whole when the run signaturesof both paragraphs match, and otherwise writes each run one unit per run.
With the edge-aware signature alone, a run that differed only by the flag
at an edge failed the shortcut while all of its units aligned as equal, so
the unchanged run was rewritten one unit per run. Bookmark ends are indexed
by run, so in a bookmarked paragraph the accept postcondition then refused
the pair at word and character granularity. The shortcut now uses
attributed_run_signature, which reads the flag the waygranular_textwrites it on a unit, so that run stays whole and keeps its original
bytes. The review proposed a wider rule, treating runs as equal whenever
their attributed unit signatures match. It was narrowed to the flag
because, with
ignore_whitespace, the wider rule lets runs with differentwhitespace take the shortcut, where
compared_run_xmlwrites the editedrun when properties differ and so breaks the left-biased ignore rule.
of a paragraph that holds bookmarks or comment ranges refuses at word and
character granularity, because the edited run is written one unit per run
and the run-indexed markers move.
mainrefusestwo wordsagainsttwo wordythere with or without the flag on both sides.mainonly letthat edit through at word granularity when the flag also differed on
edge whitespace, because every unit then differed and the whole run
became one replacement. This branch refuses that pair like the same edit
without a flag difference. In exchange, the flag-only pair that
mainrefused at character granularity in one direction now compares. Fixing
the run-splitting limit means remapping bookmark and comment markers to
the split runs, which is beyond this PR.
clear_default_orientationhelper used bysection_properties_xml,paragraph_formattingandparagraph_properties_differ. The last two compare the wholeCT_PProf acomplex paragraph, section break included. The serializer is untouched
because
default_letteranddefault_a4carrySome(Portrait), so writingit would move every
document.xmlhash baseline. The equal-section branchstill writes the original section, and a real change to landscape is still
a
section_property_change.properties. A run with nothing on the far side of the character is not
split, so a field whose
separateandendeach have their own run givesthe same bytes as before. The accept and reject postconditions read a field
as
field-owner, so the split does not reach them. The split also fixes asilent loss.
update_page_fieldswrites the result of an uncachedone-run-per-part field into the run of
end, and the redline used to carryan empty
w:inswithout the new page number. It now inserts the result. Anested field inside the result still ends the result at its own
end, asbefore. An uncached field still gets an empty
w:delin front of theinsertion, as
tracked_field_resultalways wrote, and the new test pinsthat output.
deleted_text_xmlrenames every<w:tprefix,so a
<w:tab/>inside a field result becomes<w:delTextab/>. It predatesthis PR and one-run-per-part fields hit it on
main. A packed field whosecached result holds a tab now reaches it too, where it used to fail. It
deserves its own issue: the rename should match
<w:t>,<w:tand<w:t/>only.compare()succeeds where it used to fail for aw:iddifference, for a refreshed packed field and for a default-portrait
difference on a complex section-break paragraph. It reports fewer
revisions for a flag-only
xml:spacedifference and for a default-portraitdifference, and it now shows the inserted result of a refreshed uncached
field. At word and character granularity, a whitespace unit is written
with
xml:space="preserve".try_replace_text,replace_textand regexreplacement keep
xml:space="preserve"on a rewrittenw:tthat had it.changed. It still holds with the id-only
w:sdtPrcase, since thosecontrols differ only by the id. The other rules leave every HLD statement
true.
with the debug CLI and a maturin build of rdocx-py: in the producer-trait
matrix,
w:orient="portrait"givesself=2andpacked footer fields, no cached resultgivescmpfld=ok. In the identity matrix,w:id on content controlgivescmp==0. The cells left failing belong to other PRs:empty comments part(160-3),inline content control on first runs(160-1), the TOC column (159-1),
w:tag changed(159-2b) and table-rowsaves (159-3). The review rounds changed no default-option path those cells
use.
Tests
compare_producer_noiseincrates/rdocx/tests/regression_test.rs,a named module placed next to
comparison_treats_empty_paragraph_properties_as_absent. Every case checksthe revision kinds, no diagnostics, and that accepting gives the edited side
and rejecting the original, each compared again with the same options and
no revision.
a_content_control_identity_is_not_content: the Identity attributes (w:rsid*,w14:paraId, content-controlw:id/w:tag) still breaktoc rebuild,compare()and editing: a matrix to close the class #159 TOC-shaped blockcontrol with an id added, removed or renumbered gives no revision and
keeps the original id. With one word changed inside it gives a deletion
and an insertion inside the one control. The Google Docs inline shape
(
goog_rdk_0tag andw:id) behaves the same. A control with now:sdtPragainst one with an id-onlyw:sdtPr, both ways, gives norevision and keeps the original shell.
a_rewritten_run_keeps_its_producer_space_flag: the Producer traits: a matrix over every operation, and what still fails in it and around it #160 case 2, aWORDtoWORDreplacement on a file with the flag on everyw:t,keeps the flag and compares with no revision. A replacement that removes
the edge space keeps the flag too.
the_space_flag_is_content_only_at_a_text_edge: each pair runs in bothdirections with default options, with
ignore_formatting, and at wordand character granularity. A flag-only difference without edge
whitespace, on a multiword text, or with internal double spaces gives no
revision anywhere. A flag-only difference with a trailing space gives a
deletion and an insertion on the whole-run path and nothing on the
attributed path. A Google Docs style pair (flag on every
w:t) against aWord style copy with one real edit in another paragraph gives exactly
that edit. At word and character granularity the redline writes a
split-out space as
<w:t xml:space="preserve"> </w:t>. A bookmarkedparagraph whose first run differs only by the flag on edge whitespace
gives a deletion and an insertion with default options and no revision
on the attributed path, both ways.
the_default_page_orientation_is_not_a_section_change: the Producer traits: a matrix over every operation, and what still fails in it and around it #160 case 1,a portrait file against its rdocx-edited copy, gives exactly a deletion
and an insertion. Portrait against no
w:orientgives nothing in bothdirections, and portrait against landscape gives one
SectionPropertyChange. A two-section body whose bookmarkedsection-break paragraph differs only by portrait gives nothing, and with
its text changed a deletion and an insertion, in both directions.
a_packed_field_compares_like_the_same_field_split_into_runs: the Producer traits: a matrix over every operation, and what still fails in it and around it #160section 4 script, a footer PAGE field packed or one run per part, cached
or not, compared against its copy after
update_page_fields. All foursucceed, the result and
endpart of the footer is byte-identical betweenpacked and split, and it is pinned to one deletion and one insertion that
carries the new page number.
comparison::tests::a_packed_field_result_is_read_as_one_run_per_partin
crates/rdocx/src/comparison.rs, which pins the result extraction for apacked run, a mixed run and one run per part, cached and uncached, with the
wprefix and another prefix, run attributes and run properties.placeholder::tests::replace_keeps_the_producer_space_flagincrates/rdocx-oxml/src/placeholder.rsfor the single-run and cross-runrewrite sites.
origin/mainand passes on this branch. The rowsadded in review were checked against the first version of the branch: the
id-only
w:sdtPrrow fails with "cannot revise content-controlproperties", the word and character rows fail with "comparison acceptance
does not reproduce the edited stories", and the bookmarked section-break
row fails with a formatting diagnostic. The bookmarked row added in the
second review fails on the round-2 head
13003d0fwith "comparisonacceptance does not reproduce the edited stories". Each commit builds and
passes the regression binary on its own.
w:rsid*,w14:paraId, content-controlw:id/w:tag) still breaktoc rebuild,compare()and editing: a matrix to close the class #159 section 2 and Producer traits: a matrix over every operation, and what still fails in it and around it #160 sections 2 and 4, run on theround-2 branch with the debug CLI and a maturin build of rdocx-py (the
last change only reaches the attributed path, which these scripts do not
use): the
w:idcases give 0 revisions (2 with the inner word change),case 1 gives
['deletion', 'insertion'], case 2 gives[], and everyfield row is
ok.cargo fmt --all --check,cargo clippy -p rdocx-oxml -p rdocx --all-targets --all-features -- -D warnings,python3 scripts/hash_harness.py --check(49 entries match),python3 scripts/prose_check.py,python3 scripts/sync_agent_skills.py --checkandpython3 scripts/readme_doctests.pyare clean.cargo test -p rdocx-oxml -p rdocx -p rdocx-cli --no-fail-fast: rdocx-oxml(544 lib tests and doc tests) and rdocx-cli pass, the rdocx regression
suite passes (566 tests). The failures seen are environmental only: lib
large_word_and_presentation_pdfs_preserve_logical_reading_orderandword_and_powerpoint_chart_pixels_are_identical(pinned Poppler), andintegration
odt_reader_matches_pinned_libreoffice_structure,public_authored_theme_and_fonts_match_pinned_word_resolution,sanitized_public_authoring_fixture_passes_every_conformance_stage,f267_table_style_conditional_tests::every_conditional_table_region_matches_wordand
f269_section_page_semantics::section_page_semantics_match_pinned_libreoffice_render.The last two assert the pinned LibreOffice 26.2.5.2 against the local
26.8.0.3 and fail the same way on
origin/main.ran against a maturin build of the round-2 branch: 65 passed, and the two
test_rendering_threads.pytests failed on the pinned pdfinfo version only.