Support DocBook XML sources for complete systemd documentation

[TheRealBecks]

Context

This is a follow-up to #110. I investigated adding the complete systemd documentation as systemd/systemd using the official repository:

https://github.com/systemd/systemd

A Markdown-only registry definition builds successfully, but it omits most of systemd's authoritative documentation.

Findings

The upstream documentation is split into two parts:

• docs/: 95 supported Markdown files
• man/: 484 DocBook XML files, approximately 8.1 MB

The Markdown-only prototype produced:

• 80 indexed documents
• 523 sections
• 219,498 tokens

This covers architectural and integration guides, but not the complete reference. Important documentation such as systemctl, journalctl, systemd.service, unit directives, network configuration, and most APIs lives under man/*.xml.

Context currently discovers Markdown, MDX, HTML, AsciiDoc, and reStructuredText, but not XML. Registry sources support Git and ZIP without a preprocessing step.

The XML cannot simply be treated as plain text:

• 388 files use XInclude
• 43 files reference generated entities
• 483 files declare DocBook DTDs
• upstream renders HTML using Meson and xsltproc --xinclude
• generated HTML is not committed or provided as a release asset

The official rendered documentation is available at:

https://www.freedesktop.org/software/systemd/man/latest/

However, that site has no llms.txt, and the registry cannot currently build a package by following an HTML index.

Proposed solution

My preferred solution is safe native DocBook support:

1. Recognize .xml files whose root document is a supported DocBook type such as refentry.
2. Resolve XInclude/XPointer references only within the cloned documentation root.
3. Resolve required local entities without allowing external network access or arbitrary filesystem access.
4. Convert DocBook sections, lists, tables, examples, commands, filenames, variables, and program listings into the existing parsed-document representation.
5. Add support for multiple documentation roots, for example docs and man, rather than scanning the entire source repository.
6. Add security tests for external entities, path traversal, recursive includes, entity expansion, malformed XML, and oversized documents.

A possible alternative is a constrained HTML-index source that follows same-origin links from systemd's official rendered man-page index. This would be simpler for systemd but less reproducible and would make nightly builds depend on hundreds of individual website requests.

[moshest]

Thank you for this — it's the most thorough issue this project has received. I verified your numbers rather than taking them on trust, and I'm proposing a different route than native DocBook. Reasoning below.

Your measurements

Claim	You	Measured
man/ XML size	~8.1 MB	8.13 MB	✅
files with xi:include	388	388	✅
DocBook DTD declarations	483	483	✅
XML files in man/	484	485	off by one
Markdown in docs/	95	90 .md (+5 .html)	95 is md+html
files referencing entities	43	18 files / 21 entities / 57 occurrences	doesn't reconcile

The gap you identified is real. systemctl, journalctl, systemd.service, systemd.unit and systemd.network exist only as man/*.xml — none are reachable from docs/, which is 90 architecture notes. A Markdown-only systemd package would be close to useless, exactly as you said.

Why I don't want to parse the XML

I tested linkedom (our only XML-capable dependency) against the actual files. Two things decided it:

XInclude isn't processed. xi:include survives as an inert node contributing no text — and it appears 5,036 times across those 388 files. Every page would silently lose its options, specifiers and version tables. No error, no failed build, just quietly gutted documentation. In a pipeline where one failing definition takes down the nightly publish for all 140 packages, a failure mode that doesn't fail is worse than one that does — we'd be serving confidently incomplete man pages to agents, and finding out later means re-parsing everything, not just re-fetching.

The entities can't be resolved from a checkout. All 21 are defined in man/custom-entities.ent.in — a meson template. Without running systemd's build they stay literal, so those 57 occurrences index as garbage.

One piece of good news from the test: linkedom never resolves external entities at all — <!ENTITY x SYSTEM "file:///etc/hostname"> comes back as literal text. So XXE isn't the blocker you reasonably feared. XInclude is.

What I'd rather do

freedesktop.org already publishes the rendered man pages, and they're version-pinnable:

https://www.freedesktop.org/software/systemd/man/258/   → 200
https://www.freedesktop.org/software/systemd/man/257/   → 200

The index is a real DocBook-XSL table of contents with 1,326 unique .html links, robots.txt permits man/, and .html is a format parseDocument already handles. No XML parser, no XInclude, no entity resolution, no new dependency — and pinning /man/258/ rather than /latest/ keeps builds reproducible.

What still has to be built — I want to be straight about this

Your alternative wasn't wrong to be cautious, and this isn't free:

1. The registry has no source type that can do this. SourceSchema is git | zip only (definition.ts:49), and there's no rendered-docs tarball (systemd-man.tar.gz → 404). Ingesting an HTML index means a new source type that walks a same-origin index — smaller than a DocBook parser, but still a real change with its own failure modes.
2. Your point 5 stands either way. docs_path is a single string in both schemas, so docs + man needs string | string[] (touching four call sites in build.ts) or two separate registry entries.
3. ~1,326 fetches per build needs a strategy the nightly can live with — caching, concurrency limits, and a hard rule to pin a version directory so a systemd release can't shift the build underneath us.

So: I'm accepting the goal and rejecting the mechanism. If you'd like to take the index-crawl source on I'd be glad to review it, and I'm happy to spec the schema shape first so you're not guessing at the interface. If you'd rather not, that's completely fine — this issue was valuable on its own, and correcting my assumption about the Podman rename earlier was too.

---

Generated by Claude Code