Conversation
compare() refused any pair whose content controls differed only by w:sdtPr/w:id, with "comparison cannot revise content-control properties". Word and Google Docs renumber that id when they save, so two versions of a document with a table-of-contents control could not be compared at all, although nothing in the control had changed. The id was part of control_property_signature, the one tuple that drives control alignment, the refusal and the accept and reject postconditions. Leaving it out of that tuple keeps the three consistent. Controls that differ only by id now align as unchanged and the redline keeps the original w:sdtPr bytes, and a change inside such a control is revised like any other. A w:sdtPr that holds none of the compared properties reads like no w:sdtPr, so a control that gains or loses an id-only w:sdtPr compares too. A differing w:tag, alias, control type or data binding still refuses the pair. HLD 03 now says the id is not part of the control shell. GitHub issue tensorbee#159.
A run rewritten by try_replace_text lost xml:space="preserve" on its w:t, because replace_in_single_run and replace_across_runs recomputed the flag from the new text alone. Google Docs writes the flag on every w:t, so a no-op replacement, common in a scripted pass that normalizes wording, turned an unchanged run into a deletion and an insertion once the edited copy was compared against its source. The three rewrite sites, which regex replacement shares, now keep a flag the producer wrote and still add one when the new text starts or ends with a space. compare() also read the flag as content wherever it appeared, because the run signature formatted CT_Text whole. The flag only changes how text reads when whitespace sits at an edge, so the text signature now counts it only there. The same signature serves alignment and the accept and reject postconditions, so they stay consistent, and a run that matches keeps the original bytes. Word and character granularity and the ignore options compare text units, which the redline writes back as their own w:t. Each unit copied the flag of its text, so a space split out of a flagged text differed from the same space in an unflagged one. A unit with whitespace at an edge now always carries the flag, so on that path whitespace reads as whitespace whatever the source flag was, and the flag is never content. Files from two producers that place the flag differently then compare the same way at every granularity, and a space split out of a text keeps the flag in the redline, where Word used to drop it. On that path, the shortcut that keeps runs whole when every run of a paragraph matches reads the flag the same way. Otherwise a run that differed only by the flag at an edge was rewritten one unit per run, which moved a bookmark end indexed by run and refused the pair. GitHub issue tensorbee#160.
Comparing a file whose w:pgSz carries w:orient="portrait" against its rdocx-edited copy reported a section_property_change next to the real edit. The pgSz writer emits w:orient only for landscape, so a modelled edit drops the default value, and section_properties_xml compared the two sections as whole CT_SectPr values, Some(Portrait) against None. Portrait is the schema default, so both modelled sections now read it as absent before the equality test. A paragraph that holds a bookmark, hyperlink, comment range or control compares its whole paragraph properties, section break included, so the same rule applies there. Without it such a section-break paragraph kept a formatting diagnostic, or refused the pair when its text changed. A real change to landscape is still a section property revision. The serializer is left alone, because the typed defaults of every generated document carry Some(Portrait) and writing it would move the document.xml hash baselines. GitHub issue tensorbee#160.
compare() failed with "complex field source has no end boundary" on a file against its own copy once update_page_fields had refreshed a field packed in one run, as Google Docs writes footer fields. The refreshed result lands inside that run and changes w:dirty, so compare_field_xml extracts the result on both sides, and complex_field_result looked for the end character only after the end of the run holding separate. In a packed run the end sits inside that same run. complex_field_result now splits the run so that separate ends a run and end starts one before it reads the result. The new runs repeat the original start tag and run properties, and a run with nothing on the far side of the character is left alone, so a field whose separate and end each have a run of their own yields the same bytes as before. The postconditions read a field as its owner only, so the split does not reach them. The same split fixes a quieter loss. update_page_fields writes the result of an uncached one-run-per-part field into the run of end, and the redline then carried an empty insertion without the new page number. It now inserts the result. GitHub issue tensorbee#160.
The comparison fixes, the kept xml:space flag and their tests grow the rdocx and rdocx-oxml packages, so the crates.io archive rows in README.md and crates/rdocx-oxml/README.md and their ARCHIVE_MEASUREMENTS entries are re-measured. GitHub issues tensorbee#159 and tensorbee#160.
compare() at word or character granularity refused a pair whose only difference was a word inserted or deleted beside a hyperlink or an inline control, with "comparison cannot revise paragraph boundary structures". The attributed path compared each hyperlink by the absolute number of units before its start and end, so any edit earlier in the paragraph moved both. An inline control was compared by its run index, so a word that Word writes in a run of its own before the control refused the same way, and a redline whose runs were split into units could not be compared again against the edited side. Hyperlinks and inline controls are now compared without their place in the paragraph. The units are aligned separately between consecutive shell boundaries, so the space before a control never matches the space after it, and the redline copies the original bytes before the first run of each segment, shell tags included, before anything written for that segment. Words inserted or deleted on either side of a shell, and text inserted at the start of a hyperlink, land where the edited side has them. A word moved into a hyperlink becomes a deletion outside and an insertion inside. Text inserted between two shells with no original run between them, such as before a hyperlink that opens its paragraph, still refuses the pair. The whole-run path is unchanged. GitHub issue tensorbee#161.
compare() refused any pair whose content controls differed by w:tag or w:alias, with "comparison cannot revise content-control properties". Google Docs numbers its goog_rdk_N tags per export, so two exports of one document could not be compared at all. A change to w:lock, w:placeholder or the docPartGallery was the opposite problem: it sat outside the compared properties and was dropped without a word. These five properties name, protect or file a control without changing what it holds. They leave the signature that drives alignment, refusal and the accept and reject postconditions, the redline keeps the original w:sdtPr, and each difference becomes a ComparisonDiagnostic whose message starts with "content-control <name> differs". The control type and its data binding decide what a control holds, so a difference there still refuses the pair. ComparisonDiagnostic keeps its two fields, so no struct literal breaks, and its doc comment now lists the stable message prefixes. On the word and character paths a paragraph whose runs all match now keeps them whole and compares only the controls that differ. Before, a control that differed sent the paragraph through the unit rewrite, which split unchanged runs and moved the run-indexed bookmarks Google Docs writes around headings, so the acceptance check failed. The pinned test that expected a changed tag to refuse the pair now expects the diagnostic, and HLD 03 and 10 describe the three classes. GitHub issue tensorbee#159.
compare() refused a file against its own rdocx save whenever the save wrote its comments part again, with "comments story root shell changed". The part root, with each comment or note replaced by a placeholder, and each owner start tag were compared as bytes. The XML declaration, the order of attributes and namespace declarations, a dropped declaration and an empty root written as a start and an end tag all refused the pair. A Google Docs export carries an empty self-closed comments part, and rdocx writes the w:comment attributes in another order than Word does. The root, the owner start tags and the footnote and endnote separators are now read as trees with namespace-resolved names and a sorted set of attributes. The declaration, comments, processing instructions, whitespace-only text, namespace declarations and Markup Compatibility attributes are left out, since they say how the part is written, and the redline keeps the original bytes as before. A root child or an owner attribute whose value differs still refuses the pair, so a re-dated comment keeps waiting for the redline to carry the edited comment threads. Headers and footers had no such check. Their root is not compared and their content is compared by model, which the new test pins next to the comments and notes cases. GitHub issue tensorbee#160.
compare() wraps an old field result, or a field whose instruction changed, in w:del and turns its text into deleted text. It did so by renaming every "<w:t" prefix, so a tab in the result became <w:delTextab/> and the w:textInput of a legacy text form became w:delTextextInput. The accept and reject checks read a field as its owner only, so the broken element reached the redline unseen. A refreshed result that holds a tab takes this path for a field written one run per part, and since the packed-field fix for a field packed in one run as well. The rename now matches the w:t start, end and empty tags alone. GitHub issue tensorbee#160.
The comparison changes, their doc comments and the new unit tests grow the rdocx package, so the crates.io archive row in README.md and its ARCHIVE_MEASUREMENTS entry are re-measured on top of the stacked comparison branch. GitHub issues tensorbee#159, tensorbee#160 and tensorbee#161.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Decision proposed to the maintainer: three classes of content-control
properties. A control's
w:idis identity and is ignored (done in #184).Its
w:tag,w:alias,w:lock,w:placeholderandw:docPartGalleryare metadata: the pair compares, the redline keeps the original
w:sdtPr,and each difference becomes a
ComparisonDiagnosticwhose message startswith the stable prefix
content-control <name> differs. The control typeand
w:dataBindingdecide what a control holds, so they stay structural anda difference still refuses the pair. Diagnostics keep the existing
{ location, message }struct and use message prefixes rather than a newkindfield, becauseComparisonDiagnosticis a published struct without#[non_exhaustive]and a new field would break every struct literal and thePython
__new__signature. A kind enum can come with a breaking release.With this, a Google Docs export whose
goog_rdk_Ntags were renumberedcompares with no revision and one diagnostic per control. Only the second
commit depends on this decision. The other three stand on their own.
Summary
character alignment. The attributed path compared each hyperlink by the
absolute number of units before it, so a word inserted or deleted earlier
in the paragraph refused the pair. Shells are now compared without their
place, the units are aligned separately between consecutive shell
boundaries, and the redline writes each segment after the original bytes
that open it.
refusing the pair, as proposed above. On the word and character paths, a
paragraph whose runs all match now keeps them whole and compares only the
controls that differ. The pinned test that expected a changed tag to
refuse now expects the diagnostic, and HLD 03 and 10 describe the three
classes.
comments, footnotes and endnotes parts as namespace-resolved XML trees
instead of bytes, so a part written again by another producer, rdocx
included, no longer refuses the pair.
w:twhen a compared field result is deleted. The renamematched every
<w:tprefix, so a tab in a deleted field result became<w:delTextab/>.rdocxcrates.io archive row.Part of #159. With #184 this closes section 2 (the
w:idhalf there, thew:taghalf here, andcmp==0on thew:tag changed on content controlrow of the identity matrix). Section 1 (TOC rebuild, #183), section 3
(table-row identity attributes after an edit) and the request to keep the
matrix as a test remain.
Part of #160. This fixes the compare half of section 3:
compare()of afile against its own rdocx save no longer refuses on a rewritten comments
part (
self=2on theempty comments partrow of the producer-traitmatrix). Keeping the part's bytes and
mc:Ignorableon save is #185.Section 1, the fidelity paragraph at the end of section 2 and the matrix
request remain.
Part of #161. This fixes the word and character granularity refusal for a
word inserted or deleted beside a hyperlink or an inline control, which is
the path the requested
granularityoption opens. Text inserted betweentwo shells with no original run between them still refuses, see Notes.
Exposing the options (#176 for Python, the CLI flags and default), the
comment contract and the rebuilt TOC remain.
Why
compare()refused pairs over differences that carry no content, whichthe #158 contract rules out.
compare_control_from_xmlrefused any pair whosecontrol_property_signaturediffered, and that signature held the tagand alias. Google Docs numbers
goog_rdk_Ntags per export. The lock,placeholder and gallery sat outside the signature and were dropped
without a word, which PR Compare paragraphs with identity attributes, empty properties and moved pictures #153 had noted.
compare_granular_paragraphcomparedhyperlink_shells, which countedthe attributed units before each link's start and end, and it compared
inline controls by run index. "Visit site" against "Please visit site"
refused with "cannot revise paragraph boundary structures", and so did a
word that Word writes in a run of its own before a control.
compare_owned_storycompared the comments, footnotes and endnotes root(owners replaced by a placeholder) and each owner start tag as bytes. A
Google Docs empty self-closed comments part against the open and close
pair that an rdocx save writes, or Word's
w:id-first attribute orderagainst rdocx's, refused the pair.
deleted_text_xmlusedreplace("<w:t", "<w:delText"), found in thereview of Keep producer serialization out of compare results #184.
Notes
CT_SdtPrkeeps the lock, the placeholder and the galleryas private raw slots, so
control_metadatareads them from the serializedproperties by local name, only when the two
w:sdtPrdiffer. I avoidedadding an accessor to
rdocx-oxmlso its public API does not change.Other
w:sdtPrchildren (w:showingPlcHdr,w:temporary,w15:color,list items, date format, checkbox state,
w:docPartCategory,w:docPartUnique) are still kept from the original without adiagnostic, as before. Adding one of them is one more name in
CONTROL_METADATAif you want it reported.two diagnostics at the same location, in the order tag, alias, lock,
placeholder, docPartGallery. Accepting the redline and comparing it with
the edited side gives the same diagnostics again, since the redline keeps
the original metadata. Rejecting gives the original with none.
Google Docs shape: a heading with an
h.bookmark and an inlinegoog_rdkcontrol. When a control differed, the unit rewrite split theunchanged runs around it, the run-indexed bookmark moved and the
acceptance check failed. That already happened on
mainfor a changedword inside such a control, and the same change fixes it. Output differs
from before only in that unchanged runs are no longer split into
single-unit runs.
consecutive hyperlink and inline-control boundaries, taken in document
order on both sides, so no unit matches across a shell. Without this the
space before a control could match the space after it on the edited side,
and at character granularity the
sof "see" before a link could matchthe
sof "site" inside it, which put the shell on the wrong side of theedit and refused the pair. The redline copies the original bytes before
the first run of each segment, shell tags included, before anything
written for that segment, so text inserted at the start of a segment
lands after the shell that opens it: after a hyperlink or a control, at
the start inside a hyperlink, and after a shell that ends the paragraph.
For a paragraph without a hyperlink or an inline control the alignment is
the one it was. Every existing comparison test passes unchanged. A word
moved across a shell boundary, "see" in a link and " site" after it
becoming one link "see site", used to refuse and is now a deletion outside
the link and an insertion inside it, which accepts and rejects to the two
sides.
boundaries, or the paragraph start and a boundary, have no original run
between them, because there are no original bytes to split there: before
a hyperlink or a control that opens its paragraph, between two adjacent
hyperlinks, between a control and a hyperlink right after it, and inside
an empty hyperlink.
Placing text there needs the byte position of each shell tag, which the
paragraph model does not keep, so I left it out. Shells in a different
order on the two sides refuse too. Bookmarks and comment ranges are still
compared by run index on these paths, so a paragraph with a bookmark
whose runs the word path splits fails the acceptance check, with or
without a shell, as it did before this PR. The whole-run default path is
unchanged, so a word Word writes in a run of its own before a link still
refuses at
Rungranularity.to namespaces, sorts attributes, reads an empty element as a start and an
end, and leaves out the XML declaration, comments, processing
instructions, whitespace-only text, namespace declarations and Markup
Compatibility attributes. The redline still writes the original bytes. A
root child, an owner attribute value or an owner count that differs still
refuses. Headers and footers had no such check (their root is not
compared and their content is compared by model), and the new test pins a
rewritten header root next to the comments and notes cases. Text-box host
shells inside a story are still compared as bytes. They are not related
story shells, and I left them alone to keep the diff small.
ComparisonOptions(granularity, ignored stories) incompare()#161 comment reproductions are not in these classes. A comment addedon an existing paragraph of the edited side still refuses with "cannot
revise paragraph boundary structures at body/paragraph[1]" (comment ranges
compared by run index), a comment added in a new paragraph refuses with
"comments story shell count changed", and a re-dated comment with
"comments owner shell changed", which the new test pins. They need the
redline to carry the edited side's comment anchors and threads. That
work, and the tracked replacement of the paragraphs of a rebuilt TOC, are
left to a later PR.
deleted_text_xmlnow matches thew:tstart, end and empty tags only.As before it handles the
w:prefix only, and a replaced field keepsw:instrTextinsidew:delwhere Word writesw:delInstrText. Bothpredate this PR and are left out.
ComparisonDiagnostickeeps its two fields and its doc comment now liststhe stable prefixes.
compare()andcompare_with_options()now succeed,with diagnostics, where they refused a tag or alias change. They report a
lock, placeholder or gallery change they used to drop. They succeed for a
rewritten comments or notes shell, and at word and character
granularity for a word inserted or deleted beside a hyperlink or an
inline control, text inserted at the start of a hyperlink, and a word
moved into a hyperlink. The Python stub needs no change.
no-op case no longer rewrites the part, and this PR still covers other
producers. Expose comparison options as keywords of Python Document.compare #176 exposes the granularity option in Python. The duplicate
w:sdtid note of Keep producer serialization out of compare results #184 is unchanged.shells, the shell boundaries on the attributed path and the whole-run
shortcut. 10 now describes diagnostics beyond formatting and their
prefixes.
Tests
Added. The behaviour tests failed before their fix, as quoted below. The
two refusal pins passed before and after.
granular_edits_beside_a_shell_move_its_boundaries(next togranular_hyperlink_edits_preserve_the_owner_shell), at word andcharacter granularity: sixteen pairs with a word inserted or deleted
before, after or between hyperlinks and inline controls, in the run it
extends and in a run of its own. They include the edits right next to a
shell ("Enter [control] today" to "Enter it [control] today", "see the
[control] now" to "see [control] now", "see [site], now" to "see [site]
today, now", "see [site]" to "[site]") and text after a shell that ends
the paragraph. Each compares with no diagnostic and no revision inside a
shell, keeps the shell count, and accepting and rejecting compare again
with no revision. Text inserted at the start of a hyperlink lands inside
it, a word moved into a hyperlink is deleted outside and inserted inside,
and text between two adjacent hyperlinks or before a hyperlink that opens
its paragraph still refuses with the boundary message. Before the fix
the first case refused with "cannot revise paragraph boundary
structures".
compare_producer_noisemodule of Keep producer serialization out of compare results #184:content_control_metadata_is_reported_and_the_original_kept: a blockcontrol around two paragraphs with the tag renumbered (
goog_rdk_0togoog_rdk_3, next to adocPartObjTable of Contents gallery as inIdentity attributes (
w:rsid*,w14:paraId, content-controlw:id/w:tag) still breaktoc rebuild,compare()and editing: a matrix to close the class #159 section 2), the alias, a new lock, the placeholder and the gallerychanged, each with the content unchanged (no revision) and with one
word changed (a deletion and an insertion), one diagnostic at
body/content-control[1]and the originalw:sdtPrin the redline. A tag and a lock changed together give two diagnostics.The Google Docs inline shape, with an
h.bookmark around the heading,gives one diagnostic at
body/paragraph[0]/content-control[0]at run,word and character granularity.
a_content_control_type_or_binding_change_still_refuses:w:textagainst
w:richText, and a changedw:dataBindingxpath, refuse andleave the document untouched.
a_reserialized_story_shell_is_not_a_change: the Producer traits: a matrix over every operation, and what still fails in it and around it #160 section 3 emptyGoogle Docs comments part against its rdocx no-op save and against the
rewritten form the issue describes, then a populated comments part with
rdocx's attribute order, a footnotes part with a rewritten separator and
a header with another root, with and without a comment text change.
Before the fix it refused with "comments story root shell changed".
a_changed_story_shell_still_refuses: a re-dated comment and a foreignroot child still refuse with their messages.
comparison_deletes_field_results_without_corrupting_tabs(next tocomparison_deletes_text_without_corrupting_tabs): a field result with atab, refreshed and with a new instruction. Before the fix the redline held
<w:delTextab/>.comparison.rs:only_text_elements_become_deleted_textand
an_owned_story_shell_reads_the_same_in_every_serialization.comparison_revises_nested_control_content_without_replacing_its_shellfrom the refusal to the tag diagnostic.
Run on macOS arm64:
cargo fmt --all --check,cargo clippy -p rdocx --all-targets --all-features -- -D warnings: clean.cargo test -p rdocx --no-fail-fast: lib 468 passed, integration 314passed, regression 572 passed, doc-tests 2 passed. The failures are the
known environment-only ones:
large_word_and_presentation_pdfs_preserve_logical_reading_order,word_and_powerpoint_chart_pixels_are_identical,section_page_semantics_match_pinned_libreoffice_render,every_conditional_table_region_matches_word,odt_reader_matches_pinned_libreoffice_structure,sanitized_public_authoring_fixture_passes_every_conformance_stageandpublic_authored_theme_and_fonts_match_pinned_word_resolution(pinnedtool versions).
cargo test -p rdocx-cli: 19 passed.python3 scripts/hash_harness.py --check: 49 entries match.python3 scripts/prose_check.py: 0 violations.python3 scripts/readme_doctests.py: passed after the archive re-record.run before the review changes to the word and character path, which the
matrices do not use: identity matrix,
w:tag changed on content controlgivescmp==0andcmp+1=2. The cells left failing belong to other PRs (the TOC column ofIdentity attributes (
w:rsid*,w14:paraId, content-controlw:id/w:tag) still breaktoc rebuild,compare()and editing: a matrix to close the class #159 section 1 and table-row saves of section 3). Producer-trait matrix,every cell passes,
empty comments partincluded.