Skip to content

Keep comments bytes and mc:Ignorable roots on save, and validate them - #185

Open
hadim wants to merge 5 commits into
tensorbee:mainfrom
hadim:fix/rewritten-part-roots-and-comments-part
Open

hadim wants to merge 5 commits into
tensorbee:mainfrom
hadim:fix/rewritten-part-roots-and-comments-part

Conversation

@hadim

@hadim hadim commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Maintainer decision. This PR revises the F-255 canonical comments
boundary. Every public output used to serialize an existing comments part
again, even when nothing had changed. With this PR an unchanged comments part
keeps its bytes, and a save without a comment edit no longer invalidates a
package signature over it. The recommended answer is to accept it. #140 already
gives every other modelled part this behaviour, and without it compare()
refuses any document that has a comments part against its own save. The F-255
test and HLD 04 are updated to match, with the reason stated in both.

Public API. rdocx_oxml::CT_Document gains a #[doc(hidden)] pub root_attributes: Vec<(String, String)> field. Code outside rdocx-oxml that
builds a CT_Document with a struct literal and without ..CT_Document::new()
has to add it. That is the same kind of break that background_extra_xml made
in F-X071, and I found no way around it that works: a private field would also
break the ..CT_Document::new() update syntax that rdocx itself uses, and
putting the attributes into extra_namespaces would silently change what that
documented field holds. Four struct literals in the workspace are updated. The
other API change is an addition:
rdocx_oxml::namespace::undeclared_compatibility_prefixes.

Summary

  • Keep an unchanged comments part byte for byte on save. prepare_staged_output
    (rdocx document.rs) used to mark an existing comments model dirty for every
    public output: ZIP, bytes, Flat OPC, package class, signing and encryption.
    It now marks it dirty only when the model no longer matches a fresh parse of
    its part, the same test that styles, settings and the comments-extended part
    already use. Comment mutations still set the flag themselves, as before.
  • Keep the declarations of a rewritten comments root in source order.
    CT_Comments::to_xml (rdocx-oxml comments.rs) now writes the retained root
    attributes in source order. It keeps a retained xmlns:w or xmlns:w14 in
    place, bound to the namespace the serializer writes. It adds the fixed
    declarations first only when the source lacks them, so a part rdocx authored
    serializes as before.
  • Keep mc:Ignorable on rewritten document, header and footer roots.
    CT_Document and CT_HdrFtr now keep the non-namespace attributes of their
    root in source order and write them after the namespace declarations.
    CT_HdrFtr keeps the new field private.
  • Report undeclared mc:Ignorable prefixes in rdocx validate. The new
    rdocx-oxml function lists each prefix that an mc:Ignorable or
    mc:MustUnderstand attribute names without a declaration in scope.
    validate runs it over every XML part, in part name order, and reports each
    finding as an error. The rdocx-cli README says what validate checks.
  • Re-record the archive measurements of rdocx-oxml, rdocx, rdocx-layout
    and rdocx-cli, dated 2026-09-27.

Part of #160. This PR covers the parts of section 3 that concern
serialization and validation.
Still open for other PRs: sections 1, 2, 4 and the matrix request of #160,
160-3d (compare story shells semantically, needed once a comments part really
changes), the serializer fidelity of rewritten parts (indentation,
w:val="0", redundant xmlns:w), and the same root retention for
styles.xml, which drops mc:Ignorable and its declarations when a style
is added (see Notes).

Why

The issue's section 3 script printed this on main:

parts changed by a no-op save: ['word/comments.xml']
w14 declared after: False | still in mc:Ignorable: True
compare with its own no-op save: Error: comments story root shell changed in /word/comments.xml

Three defects combine here. First, F-255 made every public output mark the
comments model dirty, so a no-op save always wrote the comments part again
through the typed model. An empty self-closed root came back as an open and
close pair, and Word's comment attribute order (w:id first) came back in
rdocx's order. compare() reads the original comments from the package bytes,
so its story skeleton check then refused a document against its own save, no-op
or edited. This hits every file with a comments part, including Word files with
real comments, not only the empty part of the #158 report fixture (which is an
open and close pair).

Second, the rewritten comments root dropped the retained xmlns:w14, and wrote
xmlns:w14 again only when some comment paragraph had a w14:paraId. It kept
mc:Ignorable="w14 w15", so the part named an undeclared prefix, which Markup
Compatibility (ECMA-376 Part 3) forbids. That still matters after the first fix
whenever the comments model really changes, for example when the last comment
is removed.

Third, CT_Document and CT_HdrFtr captured only the namespace declarations of
their root. A single try_replace_text rewrites document.xml, and the result
lost mc:Ignorable while every w14:paraId stayed in the body (the mirror case
noted in PR #154). A replacement in a header or footer did the same to that
part.

With this branch, the same script prints:

parts changed by a no-op save: []
w14 declared after: True | still in mc:Ignorable: True
compare with its own no-op save: Created 0 main-story revision element(s)

Notes

  • Decisions. The comments part boundary change is described at the top. An
    undeclared mc:Ignorable prefix is an error in validate, not a warning,
    since the part is not conformant and a consumer may reject it.
  • validate also checks mc:MustUnderstand, which is the same kind of prefix
    list with the same rule. A part whose content type does not end in xml, or
    that does not parse as XML, is outside this check. I did not add a
    well-formedness error, which would be a separate verdict. I ran it over the
    363 .docx files I had locally: python-docx outputs, the Production readiness for editing real docx and pptx files: an acceptance contract, two realistic fixtures and two matrices #158 fixtures, the
    rdocx samples, and rdocx outputs of all of them. It flags exactly the 12
    files that main's save wrote from the report fixture (the comments part with
    w14 dropped), and nothing else.
  • Root order. A rewritten comments root now keeps its full source order.
    document.xml, header and footer roots keep the order they had on main: the
    fixed w, r, mc (document only) and wp declarations first, then the
    producer declarations in source order, then the non-namespace attributes in
    source order. That keeps every prefix mc:Ignorable lists declared, since
    those rewrites already kept every producer declaration. Writing the fixed
    declarations in their source positions too would need a new representation
    of the root, and it changes nothing a consumer reads.
  • Behaviour changes:
    • A save, to_bytes, Flat OPC, package class, encrypted or signed output
      without a comment edit keeps the comments part byte for byte. Before, it
      was always written in rdocx's canonical form (fixed w: prefix, rdocx
      attribute order, XML declaration). A comment edit still writes the
      canonical form, to the part's own relationship target.
    • A no-op save of a signed package with a comments part no longer marks the
      package signature invalid.
    • Root attribute values of document.xml, headers and footers are now
      decoded when parsed. A malformed value (an undefined entity, say) now
      fails the parse. On document.xml a malformed namespace value already
      failed it. Header and footer namespace values are still read raw.
  • Found while testing, not fixed here: CT_Styles::to_xml writes only
    xmlns:w and xmlns:r. So adding a style to a Word document drops
    mc:Ignorable and every producer declaration from styles.xml, and a
    retained raw child such as <w14:ligatures w14:val="standard"/> in a style's
    run properties is left with an unbound w14 prefix. That part is then not
    namespace-well-formed. The fix is the same kind of root retention. It needs
    a new field on CT_Styles, whose fields are all public, so it is another
    break of the same kind as the one above, for a part Producer traits: a matrix over every operation, and what still fails in it and around it #160 does not report.
    I left it for a separate PR.
  • Left to other PRs:
  • Merge overlaps. rdocx document.rs changes only prepare_staged_output,
    right below the save paths that Save documents and presentations atomically, keeping links and modes #178 (atomic save) edits. rdocx-cli
    commands.rs changes only validate, which End the CLIs cleanly on a closed pipe and refuse silent overwrites #174 (CLI overwrite policy and
    closed pipe) also touches for its output. The new CLI test sits right after
    validate_exit_status_is_a_verdict. The regression tests are a named
    mod producer_part_roots_survive_save placed right after
    no_op_save_preserves_every_unchanged_part. The archive rows of rdocx,
    rdocx-oxml and rdocx-cli are re-measured by other PRs of this batch too,
    so they need one fresh measurement when the PRs are integrated.
  • The hash harness matches 49 of 49. No sample has a comments part, and the
    samples' roots carry no producer attributes.

Tests

Added:

  • rdocx regression_test.rs, mod producer_part_roots_survive_save:
    • empty_comments_part_keeps_its_bytes_and_compares_against_its_own_save:
      the issue's reproduction with its self-closed root, the fixture's open and
      close pair, and the fixture's unused default namespace. A no-op save
      changes no part, and compare against the no-op save gives no revision. A
      one-word edit keeps comments.xml byte for byte, and compare against it
      gives a deletion and an insertion.
    • word_comments_keep_their_bytes_through_a_body_edit: the same checks with
      a Word-style comment (w:id first, w14:paraId on its paragraph) and its
      anchors in the body.
    • rewritten_comments_root_keeps_every_declaration_in_source_order:
      removing the last Word comment rewrites the part, which must keep the
      source root start tag and declare w14 and w15.
    • edited_main_document_keeps_mc_ignorable_while_its_body_uses_w14: after a
      one-word edit, document.xml keeps mc:Ignorable="w14 w15" with both
      prefixes declared, the untouched paragraph keeps its w14:paraId, compare
      gives a deletion and an insertion, and a second edit keeps the same root.
    • rewritten_header_and_footer_keep_mc_ignorable_and_its_declarations: a
      replacement that hits a header and a footer keeps mc:Ignorable and its
      declarations on both roots.
  • rdocx integration_test.rs, comments_part_uses_its_existing_relationship_target
    (F-255): the output without a comment edit keeps the custom-target part byte
    for byte, leaves the signature valid, and the Flat OPC keeps <x:comments.
    The canonical w: root, the raw children, the custom target, the signature
    invalidation and the ZIP and Flat OPC agreement are now checked after
    add_comment, which is when the model is written.
  • rdocx-oxml unit tests: comments::tests::rewritten_root_keeps_its_declarations_in_source_order,
    document::tests::root_attributes_survive_a_rewrite_after_the_namespace_declarations,
    header_footer::tests::root_attributes_survive_a_rewrite_after_the_namespace_declarations
    and namespace::tests::compatibility_prefixes_must_be_declared_in_scope.
  • rdocx-cli tests/integration.rs: validate_reports_an_undeclared_ignorable_prefix,
    on the comments part that main's save wrote from the report fixture.

On main, the five regression tests and the CLI test fail, and so does the
comments unit test against the old serializer. The document and header unit
tests read the new field, so they do not compile on main.

End to end, with a debug build of this branch:

  • The issue's section 3 script prints the output shown above.
  • matrix_producer_traits.py from Production readiness for editing real docx and pptx files: an acceptance contract, two realistic fixtures and two matrices #158: the row empty comments part now
    passes cmpfld and self (both FAIL on main). The other rows are
    unchanged. Their failing cells belong to other PRs.
  • On fixture-report.docx: a no-op rdocx replace changes no part, and a
    one-word replacement changes only document.xml, which keeps
    mc:Ignorable with all eight prefixes declared. rdocx compare of the
    fixture against both outputs succeeds with no diagnostic, and
    rdocx validate reports no markup compatibility error on the fixture or on
    either output.

Run on macOS arm64, debug build, at the head of the branch:

  • cargo fmt --all --check and
    cargo clippy -p rdocx-oxml -p rdocx -p rdocx-layout -p rdocx-cli --all-targets --all-features -- -D warnings:
    clean. Each of the four fix commits was also checked on its own with clippy
    and its tests.
  • cargo test -p rdocx-oxml: 547 passed, plus 1 doctest.
  • cargo test -p rdocx --no-fail-fast: regression 566 passed, doctests 2
    passed. Lib and integration have only the environment failures listed
    below.
  • cargo test -p rdocx-layout: 292 passed, plus 1 doctest.
  • cargo test -p rdocx-cli: 2 unit and 18 integration tests passed.
  • cargo check -p rdocx-py -p rdocx-wasm --all-targets and
    cargo check --target wasm32-unknown-unknown -p rdocx-wasm: clean.
  • RUSTDOCFLAGS="-D warnings" cargo doc -p rdocx-oxml -p rdocx --no-deps --all-features: clean.
  • python3 scripts/hash_harness.py --check: 49 entries match.
  • python3 scripts/prose_check.py: 0 violations, also on every commit
    message.
  • python3 scripts/readme_doctests.py: passes with the re-recorded rows.
  • No Python binding or stub changed, so the Python gate was not run. The
    binding was built only to run the issue's scripts.

Environment failures seen here, all on the known list:

  • rdocx lib: large_word_and_presentation_pdfs_preserve_logical_reading_order
    and word_and_powerpoint_chart_pixels_are_identical (pinned Poppler).
  • rdocx integration, pinned LibreOffice and Word oracles:
    odt_reader_matches_pinned_libreoffice_structure,
    public_authored_theme_and_fonts_match_pinned_word_resolution,
    section_page_semantics_match_pinned_libreoffice_render,
    every_conditional_table_region_matches_word, and
    sanitized_public_authoring_fixture_passes_every_conformance_stage, which
    needs an offline cargo run.

Every public output (save, to_bytes, Flat OPC, package class, signing
and encryption) marked an existing comments model dirty before staging,
so a save without any comment edit serialized the comments part again
through the typed model. An empty self-closed root came back as an open
and close pair, and Word's comment attribute order changed. Compare
reads the original comments from the package bytes, so it then refused
a document against its own save with "comments story root shell
changed".

The output boundary now marks the model dirty only when it no longer
matches a fresh parse of its part, as styles, settings and the
comments-extended part already do. An unchanged part keeps its bytes,
which also leaves a package signature over it valid. This revises the
F-255 canonical comments boundary. The F-255 test now pins the
byte-identical output without a comment edit, and keeps its canonical
rewrite, custom target and Flat OPC checks behind a comment edit.

GitHub issue tensorbee#160.
CT_Comments::to_xml wrote xmlns:w first, skipped the retained xmlns:w
and xmlns:w14 declarations, and wrote xmlns:w14 again only when a
comment paragraph carried a paragraph id. A rewritten part without one,
such as a Word comments part after its last comment is removed, kept
mc:Ignorable="w14 ..." without declaring w14, which Markup
Compatibility forbids.

The root now writes its retained attributes in source order. A retained
w or w14 declaration stays where it stands and binds the namespace the
serializer writes. The fixed declarations come first only when the
source lacks them, so a part rdocx authored serializes as before.

GitHub issue tensorbee#160.
CT_Document and CT_HdrFtr captured only the namespace declarations of
their root. Any typed rewrite of document.xml, which one replacement is
enough to cause, or of a header or footer therefore dropped
mc:Ignorable while the prefixes it lists stayed declared and in use,
such as w14:paraId on every paragraph Word writes.

Both models now keep the other root attributes in source order and
write them after the namespace declarations. Every prefix such an
attribute lists stays declared, since the rewrite already keeps every
producer declaration. CT_Document gains a hidden public
root_attributes field, as background_extra_xml was added, so a struct
literal outside rdocx-oxml needs the new field. CT_HdrFtr already has
a private field and keeps the new one private.

GitHub issue tensorbee#160.
rdocx validate checked relationships and content types, plus advisory
findings, so it passed the comments part that earlier saves wrote with
w14 listed in mc:Ignorable and no w14 declaration. Markup Compatibility
requires every listed prefix to be declared, and a consumer may reject
such a part.

rdocx-oxml gains undeclared_compatibility_prefixes, which reports each
prefix that an mc:Ignorable or mc:MustUnderstand attribute lists
without a declaration in scope. validate runs it over every XML part in
part name order and reports each finding as an error.

GitHub issue tensorbee#160.
The fixes on this branch change the packaged sources of rdocx-oxml,
rdocx, rdocx-layout and rdocx-cli, and the rdocx-cli README, so their
crates.io archive rows are measured again. The four rows are dated
2026-09-27 through ARCHIVE_REMEASUREMENT_DATES, as the latest
re-measurements on main are.

GitHub issue tensorbee#160.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant