Skip to content

docs: serve the newest release at user-guide/latest and the main snapshot at user-guide/dev #7684

Open

Description

@andygrove

What is the problem?

https://datafusion.apache.org/comet/user-guide/latest/ serves the unreleased docs built from main. There is no stable URL for the newest release, and several places send users to latest/ expecting release docs:

  • CometConf.COMPAT_GUIDE and CometConf.TUNING_GUIDE put user-guide/latest/... links into config descriptions and fallback messages, so a released 1.2 jar sends users to docs that describe unreleased behavior.
  • The home page buttons link to user-guide/latest/....
  • The user guide index lists the main-branch guide as "Development Snapshot <latest/index>".

Apache Spark and Apache Iceberg serve the newest release at latest/, which is what users expect from that name.

Proposal

URL Content
user-guide/latest/ The newest release
user-guide/dev/ The development snapshot, built from main
user-guide/1.2/ Each release, pinned (as today)

Only the versioned URLs are permanent. Pages under latest/ and dev/ may be added, renamed or removed between releases, and we do not add redirects to keep old latest/ deep links alive. The user guide index should say so. Expected consequences: existing links to /user-guide/latest/... start showing release content, and pages that exist only on main will 404 under latest/ until the next release.

Prior work

I opened #7112 for this. It has one approval and the review comments were addressed, but I do not have time to finish it. It is 109 files and now conflicts with main. About 50 of those files are the mechanical latest to dev source directory rename, so it is easier to redo on current main than to resolve the conflicts. The branch is andygrove:documentation-publishing-54d88243 and still has the working implementation. Its diff also carries unrelated hunks from an older base (for example the Spark 3.4 profile and the release build notes), so take only the docs and publishing changes.

Design that worked

  1. Rename the source directory docs/source/user-guide/latest to dev. This is required, not cosmetic. Sphinx resolves the contributor guide's ../user-guide/latest/... links by docname, and the latest alias cannot also be a source directory. About 70 files outside the directory reference it, so grep for user-guide/latest after the rename. Known spots: .github/labeler.yml, .prettierignore, dev/generate-release-docs.sh, docs/source/conf.py (the exclude_patterns entry), docs/README.md, contrib/delta-spark/README.md, the contributor guide pages, the expression audit pages, spark/pom.xml (the generate-docs output paths), and the agent skills under .ai/skills/ (wire-datafusion-function, review-comet-expression-pr).
  2. docs/generate-versions.py.
    • Move the release versions to two constants at the top of the file (LATEST_RELEASED_VERSION, PREVIOUS_VERSIONS), so promoting a release also moves latest.
    • Build the main-branch guide as dev.
    • Detect each release branch's guide directory instead of switching on the version number. branch-1.1 and branch-1.2 both keep it under user-guide/latest/, and branches cut after the rename will use user-guide/dev/. Raise if neither exists, since falling back to the whole user-guide tree hides a failed git clone.
    • Fail the build if docs/source/user-guide/latest exists on main, because the alias step deletes whatever Sphinx built there.
  3. Alias after the Sphinx build. Add generate-versions.py --alias-latest <html dir>, called from docs/build.sh after Sphinx, which copies user-guide/<major.minor>/ to user-guide/latest/. Copy the finished HTML instead of building the sources twice, so Sphinx sees each document once. Use a copy, not a symlink, because the site is published by rsync into asf-site and ASF's web server is not known to follow symlinks. Every link in the pages is relative, so the copy works one directory over.
  4. Redirects in docs/source/conf.py. The five user-guide/latest/... redirects covered old dev-tree URLs and should go. 1.2 has real pages for compatibility/scans.html, tuning/scans.html and iceberg-writes.html, but not for compatibility.html or compatibility/parquet-iceberg-known-issues.html, so those two will 404 under latest/ until the next release. sphinx-reredirects writes its stub pages during html-collect-pages, and the alias step then removes the whole latest/ directory, so keeping a redirect entry has no effect as written. If we want to keep those two, copy with shutil.copytree(source, target, dirs_exist_ok=True) instead of deleting the target first (build.sh already starts from an empty build/).
  5. Version-aware links in the jar. Add a small CometDocs object that maps the running Comet version to a guide directory: 1.2.3 gives 1.2, and a snapshot or unknown version gives dev. Build COMPAT_GUIDE and TUNING_GUIDE from it. The plan stability suites embed these URLs in the TPC-DS goldens (q24a, q24b and q24 extended.txt), so CometPlanStabilitySuite has to normalize the version to {version} or every release bump rewrites the goldens. Add a CometDocsSuite for the version mapping.
  6. Docs. Update the user guide index (a URL table, the "current stable release" sentence, the toctree entry), docs/README.md, release_process.md (promoting a release now means editing the two constants, and index.md has a new URL-table row to update) and backporting.md.

Review findings to carry over

  • CometDocsSuite must be registered in .github/workflows/pr_build_linux.yml. python3 dev/ci/check-suites.py exits non-zero otherwise.
  • The agent skill instructions (.ai/skills/wire-datafusion-function/SKILL.md and the others) must point at user-guide/dev/ in the same change. Following the old text recreates the latest source directory that the guard rejects.
  • PR CI never runs docs/build.sh. The docs job runs only on push to main, so the first CI run of the alias step is the post-merge deploy. A failure there stops before rsync, leaving the site stale rather than broken, but I would still run the full docs/build.sh locally first.
  • After the rename, iceberg-writes.md no longer exists (it was merged into iceberg.md), so the benchmark source comments that cite it should point at the "Accepted divergences" section of dev/iceberg.md. Also run make format so spotless is happy with the rewrapped comment lines.
  • The backporting text must not claim that 1.2 and later branches use dev. branch-1.2 was cut before the rename and has latest. Say that branches cut after the rename use dev, and that git's rename detection usually carries a cherry-pick across it. The release branch freeze script, dev/generate-release-docs.sh, also has to work on both names.
  • Mermaid diagrams need mmdc to render. I had it missing locally, so check that CI or the first publish draws them.

How I validated it

  • Ran the full docs/build.sh (clones the release branches, the generate-docs Maven profiles, Sphinx, then the alias step). latest/ was byte-identical to the newest release directory (diff -rq).
  • Checked about 165,000 href and src attributes in the built HTML for existence. The only misses were unrendered theme template fragments in _static/webpack-macros.html.
  • Checked that user-guide/installation.html still redirects to latest/installation.html, that the home page hero links resolve, and that a page under latest/ links to ../dev/index.html from the sidebar.
  • Checked that no conditional block markers or literal $COMET_VERSION leaked into dev/ or latest/.
  • Not checked: how the sidebar renders and highlights from a page under latest/. Look at that on the first publish.

Suggested approach for whoever picks this up

  1. Do the rename with git mv on current main in a quiet period. It touches about 120 files, so it conflicts with any open docs PR. Announce it on the PR or dev list first.
  2. Set LATEST_RELEASED_VERSION to the newest release at that time (1.2.0 now) and update index.md and older-versions.md to match.
  3. Reapply the generator, conf.py, CometDocs, plan stability and docs changes from the branch above.
  4. Run docs/build.sh locally and repeat the link check.
  5. Open the PR with the documentation label.

Acceptance criteria

  • user-guide/latest/ serves the newest release and user-guide/dev/ serves main.
  • A released jar's config descriptions and fallback messages link to its own user-guide/<major.minor>/ pages, and a snapshot build links to dev.
  • The user guide index states that only versioned URLs are stable.
  • Promoting a release at release time is a one-place change in generate-versions.py, documented in release_process.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions