chore(draft): setup github action for generating editor's draft .html/ .docx#11
chore(draft): setup github action for generating editor's draft .html/ .docx#11awoie wants to merge 13 commits into
Conversation
|
(feedback from claude/codex, and '1' is missing as it wasn't helpful so I deleted it) 2. HTML draft numbers ISO front matter as clauses; DOCX does notFile:
Fix: Make the HTML rendition unnumber the front matter too (e.g. frontmatter 3. mmark-to-pandoc line filters are not fenced-code-block awareFile: The filters that drop Fix: Track fence state ( 4. Abstract-dropping swallows any lower-level headings after
|
- mmark-to-pandoc.py: parse TOML with tomllib, fence-aware filtering, abstract drop stops at any heading, title/status as separate fields (#3,4,6,7) - add golden-output test for the converter (tests/) - commit template/iso-reference.docx; drop generator from build path (#5) - make-iso-reference.py: drop word/embeddings/, strip dangling drawing refs (#8) - Makefile: clean before build, PYTHON override, pin markdown2rfc by digest (#9,13) - build.yml: pages concurrency group, pinned pandoc .deb, pin Python 3.12 (#10,11) - draft: unnumbered Foreword/Introduction so Scope=clause 1 in both outputs (#2); add seriesInfo stream (#9); relabel Working Group Draft -> Editor's Copy (#12)
Thanks for the detailed review. Addressed in the latest push (AI-assisted). Converter (parse, don't pattern-match).
Reference docx. Generated once and committed as Build/CI. finding 13 clean-before-build; finding 9 Findings 2 + 12. Foreword/Introduction are now unnumbered front matter ( Not going the Metanorma route: it would mean AsciiDoc and dropping the |
- seriesInfo stream independent conflicted with xml2rfc's default submissionType (IETF) and failed # Clean stale intermediates first so the copy below is unambiguous even if a # previous run was interrupted. rm -f draft/digital-credentials-harmonized-presentation*.html draft/digital-credentials-harmonized-presentation*.xml draft/digital-credentials-harmonized-presentation*.txt docker run --rm -v "/Users/oliver.terbu/git/openid/dchp/draft:/data" danielfett/markdown2rfc@sha256:7b4412559d6ba5db45a14174a28da5b240512e7c2a886a5e4adb44e5e67f34ca digital-credentials-harmonized-presentation.md Dockerfile danielfett/markdown2rfc Using versions: mmark: 2.2.31 xml2rfc 3.16.0 --------------------- mkdir -p build # Cleaned above, so this glob now matches exactly the fresh output whatever # markdown2rfc names it (it may add a draft-version suffix). cp draft/digital-credentials-harmonized-presentation*.html build/digital-credentials-harmonized-presentation-editors-copy.html rm -f draft/digital-credentials-harmonized-presentation*.html draft/digital-credentials-harmonized-presentation*.xml draft/digital-credentials-harmonized-presentation*.txt HTML Editor's Copy -> build/digital-credentials-harmonized-presentation-editors-copy.html; set stream = IETF to clear the empty-stream warning without the conflict (finding 9) - Makefile: auto-detect a tomllib-capable Python (3.11+) so mkdir -p build python3.13 tools/mmark-to-pandoc.py < draft/digital-credentials-harmonized-presentation.md > build/digital-credentials-harmonized-presentation.pandoc.md pandoc build/digital-credentials-harmonized-presentation.pandoc.md \ --reference-doc=template/iso-reference.docx \ --lua-filter=tools/iso-styles.lua \ -o build/digital-credentials-harmonized-presentation.docx rm -f build/digital-credentials-harmonized-presentation.pandoc.md ISO Word document -> build/digital-credentials-harmonized-presentation.docx works where the default python3 is older, with a clear error if none is found - Makefile: revert the HTML copy to a glob (markdown2rfc may add a draft-version suffix); the clean-before-build step keeps it unambiguous (finding 13)
Follow-up: fixed two build issues from the first pass (AI-assisted).
|
|
It seems there is a build error on github. I will fix and let you know once fixed @jogu . |
awoie
changed the title
mmark hard-codes the rfc submissionType as IETF and won't accept a non-IETF stream, so the previous stream = IETF was misleading and stream = independent fails the build. Leave stream unset (benign Empty-stream warning, HTML still builds) and note that the real publication reference is a WG/SDO decision (issue 13).
|
Would it make sense to move the tooling to a separate repository in OIDF and provide it ready to use as a container? |
The build job runs tests/test_mmark_to_pandoc.py, but the push and pull_request paths filters did not include tests/**, so a PR touching only the test or its golden fixtures merged with no CI run at all; a fixture/converter mismatch would only surface on the next unrelated PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The committed template/iso-reference.docx shipped three things from the
ISO Word template that the generator's docstring promised were stripped:
* docProps/core.xml with the template author's personal metadata
(dc:creator / cp:lastModifiedBy) — a third party's personal data in
a public repo; only app.xml and custom.xml were dropped. Drop the
whole docProps/ tree (Word and pandoc do not need document
properties) and strip the package references to it.
* the entire ISO template body: title page, WD/CD review warning, and
the full ISO copyright/IPR notice (© ISO, "All rights reserved…",
the ISO copyright office address). Pandoc ignores the reference
document's body — it reads the styles and the body-level <w:sectPr>
only — so replace the body with one empty paragraph plus that
sectPr, and strip drawing/object elements from the kept
headers/footers instead (their media relationships were already
being removed).
* [trash]/0000.dat and [trash]/0001.dat — zip-housekeeping junk left
in the .dotx by whatever tool last edited the ISO template.
Add tests/test_iso_reference.py pinning all of this against the
committed file (docProps absence, ISO-boilerplate tripwires, junk
entries, sectPr preservation) and run it in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fence tracking normalised every fence to three characters and
accepted any indentation, so three distinct kinds of legal fence content
desynced the open/closed state and mmark filtering ran inside verbatim
examples (or, inverted, {backmatter}/{: ...} lines leaked into the Word
document as literal text):
* a 3-backtick line inside a 4+-backtick fence was taken as the closer
(CommonMark: a closer must be at least as long as the opener);
* a '```lang' line inside an open fence was taken as the closer
(closing fences must be bare — no info string);
* backticks indented 4+ spaces (indented-code content) toggled fence
state (fences may be indented at most 3 spaces).
Track the opener's marker verbatim and require closers to match its
character, meet its length, and carry no info string; also reject
backtick openers whose info string contains a backtick. Each rule is
pinned by a new focused test case.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three independent ways the %%% ... %%% detection deviated from how
mmark reads the file, each leaking or crashing instead of converting:
* a UTF-8 BOM (e.g. from Notepad) hid the opening delimiter —
str.strip() does not remove U+FEFF — so the raw TOML (title,
seriesInfo, author emails) was emitted verbatim as Word body text;
* a '%%%' line inside a TOML multi-line string was taken as the
closing delimiter, so tomllib crashed on the truncated block even
though mmark parses the same file fine;
* an unclosed block was silently treated as body, again rendering the
raw TOML into the Word document.
Strip a leading BOM, find the closing delimiter by parsing (a candidate
that leaves invalid TOML behind is not the closer), and raise a clear
error for unclosed/invalid front matter so `make docx` fails loudly
rather than shipping a garbled deliverable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The abstract-dropping state only terminated on ATX headings ('#'/'.#')
or part markers, so a setext-style heading after .# Abstract
Foreword
========
was swallowed together with its entire section body — whole sections
silently vanished from the Word document. Look one line ahead for a
=/- underline under a non-blank line and end the drop there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ABSTRACT_START and IAL_LINE were anchored at column 0 while the heading
regexes tolerate up to 3 leading spaces (per CommonMark/kramdown). An
abstract heading indented 1-3 spaces was therefore rewritten to a
literal '# Abstract' clause — shipping the abstract into an ISO
deliverable that must not have one — and an indented attribute-list
line (e.g. attached to a list item) rendered as literal '{: ...}' body
text in the Word document.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
convert() used str.splitlines(), which also breaks lines at U+2028, U+0085, form feed, and friends — characters markdown (and mmark) treat as ordinary text. A LINE SEPARATOR pasted into prose from Word or a PDF (a common artifact when copying from ISO documents) inside a heading or table row therefore split that line in two in the Word rendition only, silently diverging from the HTML rendition. Normalise CRLF/CR and split on "\n" alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
template/Word_template_for_ISO_standards.dotx carried everything the derived reference docx scrubs: the template author's personal metadata (docProps/core.xml), an "ISO - Internal" classification label (docProps/custom.xml), the full ISO copyright/title-page/footer boilerplate, the ISO logo/OLE/customXml parts, and [trash]/ zip junk — and it is ISO-copyrighted material in a public repo. Sanitizing the committed iso-reference.docx while publishing its dirty source defeated the point. Drop the .dotx; the committed iso-reference.docx remains the build input. To regenerate it, obtain the template from ISO/IEC JTC 1/SC 17 WG 10 and place it at the documented path (the generator now writes to the committed location by default — the old default, build/, was read by nothing). Extend tests/test_iso_reference.py to guard the committed artifacts: the hygiene checks scan every committed .docx/.dotx (via git ls-files, with a new "ISO - Internal" tripwire), and a sync check rebuilds the reference document in memory and compares it part-for-part (decompressed, so zlib differences between environments cannot cause false failures) so a template or generator change cannot silently ship a stale docx — it runs whenever the ISO template is present locally and skips otherwise. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Makefile: add a `make test` target (globs tests/test_*.py, so a new
test runs in CI without editing the workflow) and share the
Python-with-tomllib guard between docx and test via a need-python
prerequisite; build.yml now runs `make test` like the other steps
delegate to make.
- mmark-to-pandoc.py: spell the 3-space block-indent tolerance the same
way in all seven regexes; this also fixes the last members of the
indented-construct leak class (an indented {mainmatter} passed
through as literal Word text, tab-indented headings ended the
abstract drop). Test case added.
- make-iso-reference.py: derive _DANGLING from DROP_PREFIXES so the
drop list and the reference-stripping list cannot drift (verified
output-identical against the ISO template); inline the trivial
_is_dropped wrapper.
- tests: read each zip part once in the boilerplate scan instead of
once per marker; open the reference docx once in main(); drop the
redundant name parameter threaded into diff headers; replace
repeated why-the-template-is-not-committed prose with pointers to
the generator docstring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 participants
This PR does the following:
Fixes #12