Repository navigation
Description
What is the problem?
https://datafusion.apache.org/comet/user-guide/latest/ serves the unreleased docs built from main. There is no stable URL for the newest release, and several places send users to latest/ expecting release docs:
CometConf.COMPAT_GUIDEandCometConf.TUNING_GUIDEputuser-guide/latest/...links into config descriptions and fallback messages, so a released 1.2 jar sends users to docs that describe unreleased behavior.- The home page buttons link to
user-guide/latest/.... - The user guide index lists the main-branch guide as "Development Snapshot <latest/index>".
Apache Spark and Apache Iceberg serve the newest release at latest/, which is what users expect from that name.
Proposal
| URL | Content |
|---|---|
user-guide/latest/ |
The newest release |
user-guide/dev/ |
The development snapshot, built from main |
user-guide/1.2/ |
Each release, pinned (as today) |
Only the versioned URLs are permanent. Pages under latest/ and dev/ may be added, renamed or removed between releases, and we do not add redirects to keep old latest/ deep links alive. The user guide index should say so. Expected consequences: existing links to /user-guide/latest/... start showing release content, and pages that exist only on main will 404 under latest/ until the next release.
Prior work
I opened #7112 for this. It has one approval and the review comments were addressed, but I do not have time to finish it. It is 109 files and now conflicts with main. About 50 of those files are the mechanical latest to dev source directory rename, so it is easier to redo on current main than to resolve the conflicts. The branch is andygrove:documentation-publishing-54d88243 and still has the working implementation. Its diff also carries unrelated hunks from an older base (for example the Spark 3.4 profile and the release build notes), so take only the docs and publishing changes.
Design that worked
- Rename the source directory
docs/source/user-guide/latesttodev. This is required, not cosmetic. Sphinx resolves the contributor guide's../user-guide/latest/...links by docname, and thelatestalias cannot also be a source directory. About 70 files outside the directory reference it, so grep foruser-guide/latestafter the rename. Known spots:.github/labeler.yml,.prettierignore,dev/generate-release-docs.sh,docs/source/conf.py(theexclude_patternsentry),docs/README.md,contrib/delta-spark/README.md, the contributor guide pages, the expression audit pages,spark/pom.xml(thegenerate-docsoutput paths), and the agent skills under.ai/skills/(wire-datafusion-function,review-comet-expression-pr). docs/generate-versions.py.- Move the release versions to two constants at the top of the file (
LATEST_RELEASED_VERSION,PREVIOUS_VERSIONS), so promoting a release also moveslatest. - Build the main-branch guide as
dev. - Detect each release branch's guide directory instead of switching on the version number.
branch-1.1andbranch-1.2both keep it underuser-guide/latest/, and branches cut after the rename will useuser-guide/dev/. Raise if neither exists, since falling back to the wholeuser-guidetree hides a failedgit clone. - Fail the build if
docs/source/user-guide/latestexists on main, because the alias step deletes whatever Sphinx built there.
- Move the release versions to two constants at the top of the file (
- Alias after the Sphinx build. Add
generate-versions.py --alias-latest <html dir>, called fromdocs/build.shafter Sphinx, which copiesuser-guide/<major.minor>/touser-guide/latest/. Copy the finished HTML instead of building the sources twice, so Sphinx sees each document once. Use a copy, not a symlink, because the site is published by rsync intoasf-siteand ASF's web server is not known to follow symlinks. Every link in the pages is relative, so the copy works one directory over. - Redirects in
docs/source/conf.py. The fiveuser-guide/latest/...redirects covered old dev-tree URLs and should go. 1.2 has real pages forcompatibility/scans.html,tuning/scans.htmlandiceberg-writes.html, but not forcompatibility.htmlorcompatibility/parquet-iceberg-known-issues.html, so those two will 404 underlatest/until the next release. sphinx-reredirects writes its stub pages duringhtml-collect-pages, and the alias step then removes the wholelatest/directory, so keeping a redirect entry has no effect as written. If we want to keep those two, copy withshutil.copytree(source, target, dirs_exist_ok=True)instead of deleting the target first (build.shalready starts from an emptybuild/). - Version-aware links in the jar. Add a small
CometDocsobject that maps the running Comet version to a guide directory:1.2.3gives1.2, and a snapshot or unknown version givesdev. BuildCOMPAT_GUIDEandTUNING_GUIDEfrom it. The plan stability suites embed these URLs in the TPC-DS goldens (q24a,q24bandq24extended.txt), soCometPlanStabilitySuitehas to normalize the version to{version}or every release bump rewrites the goldens. Add aCometDocsSuitefor the version mapping. - Docs. Update the user guide index (a URL table, the "current stable release" sentence, the toctree entry),
docs/README.md,release_process.md(promoting a release now means editing the two constants, andindex.mdhas a new URL-table row to update) andbackporting.md.
Review findings to carry over
CometDocsSuitemust be registered in.github/workflows/pr_build_linux.yml.python3 dev/ci/check-suites.pyexits non-zero otherwise.- The agent skill instructions (
.ai/skills/wire-datafusion-function/SKILL.mdand the others) must point atuser-guide/dev/in the same change. Following the old text recreates thelatestsource directory that the guard rejects. - PR CI never runs
docs/build.sh. Thedocsjob runs only on push tomain, so the first CI run of the alias step is the post-merge deploy. A failure there stops before rsync, leaving the site stale rather than broken, but I would still run the fulldocs/build.shlocally first. - After the rename,
iceberg-writes.mdno longer exists (it was merged intoiceberg.md), so the benchmark source comments that cite it should point at the "Accepted divergences" section ofdev/iceberg.md. Also runmake formatso spotless is happy with the rewrapped comment lines. - The backporting text must not claim that 1.2 and later branches use
dev.branch-1.2was cut before the rename and haslatest. Say that branches cut after the rename usedev, and that git's rename detection usually carries a cherry-pick across it. The release branch freeze script,dev/generate-release-docs.sh, also has to work on both names. - Mermaid diagrams need
mmdcto render. I had it missing locally, so check that CI or the first publish draws them.
How I validated it
- Ran the full
docs/build.sh(clones the release branches, thegenerate-docsMaven profiles, Sphinx, then the alias step).latest/was byte-identical to the newest release directory (diff -rq). - Checked about 165,000
hrefandsrcattributes in the built HTML for existence. The only misses were unrendered theme template fragments in_static/webpack-macros.html. - Checked that
user-guide/installation.htmlstill redirects tolatest/installation.html, that the home page hero links resolve, and that a page underlatest/links to../dev/index.htmlfrom the sidebar. - Checked that no conditional block markers or literal
$COMET_VERSIONleaked intodev/orlatest/. - Not checked: how the sidebar renders and highlights from a page under
latest/. Look at that on the first publish.
Suggested approach for whoever picks this up
- Do the rename with
git mvon currentmainin a quiet period. It touches about 120 files, so it conflicts with any open docs PR. Announce it on the PR or dev list first. - Set
LATEST_RELEASED_VERSIONto the newest release at that time (1.2.0 now) and updateindex.mdandolder-versions.mdto match. - Reapply the generator,
conf.py,CometDocs, plan stability and docs changes from the branch above. - Run
docs/build.shlocally and repeat the link check. - Open the PR with the
documentationlabel.
Acceptance criteria
user-guide/latest/serves the newest release anduser-guide/dev/serves main.- A released jar's config descriptions and fallback messages link to its own
user-guide/<major.minor>/pages, and a snapshot build links todev. - The user guide index states that only versioned URLs are stable.
- Promoting a release at release time is a one-place change in
generate-versions.py, documented inrelease_process.md.
Activity