Add versioned documentation (litellm 1.79.0 → 1.87.1)#305
Open
yuneng-berri wants to merge 14 commits into
Open
Add versioned documentation (litellm 1.79.0 → 1.87.1)#305yuneng-berri wants to merge 14 commits into
yuneng-berri wants to merge 14 commits into
Conversation
…sions) Implements Docusaurus versioning for litellm-docs: - versioning/build_manifest.py: maps each final pip release (>=1.79.0) to the docs-repo commit current at its PyPI publish time (deterministic, reproducible) - versioning/generate_versions.sh: idempotent backfill that snapshots historical docs/ + sidebars.js per release via 'docusaurus docs:version' - docusaurus.config.js: lastVersion=newest release at /docs/, current tree as 'main' at /docs/main/, version dropdown + 'All versions' link; degrades gracefully when no versions are generated - plugins/versioned-seo: post-build noindex + canonical-to-latest on all non-latest pages (avoids duplicate-content dilution, scopes search to latest) - src/pages/versions.js: /versions listing with live version data + release dates - versioning/README.md: methodology, date-mapping caveat, regen instructions Generated versioned_docs/ snapshots are committed separately.
Historical docs reference repo-root images relatively (../../img/foo.png), which break inside versioned_docs/version-X/ (one level deeper than docs/). Expose versioned_docs/img as a symlink to ../img so all still-present images resolve across every version, and add fill_missing_images.py to restore images removed from img/ since (referenced by old snapshots). Restores enterprise_vs_oss.png.
Generated via versioning/generate_versions.sh: one Docusaurus version per final
pip release from 1.79.0 (2025-10-26) through 1.87.1, each snapshotting docs/ +
sidebars.js from the release's mapped source commit.
- versioned_docs/version-*/ : 73 doc snapshots
- versioned_sidebars/ : per-version sidebars
- versions.json : version list (newest first), drives lastVersion
- versioned_docs/{img,src,static}: symlinks to repo-root siblings so escaping
relative refs resolve across all versions
- versioning/link_escaping_siblings.py: data-driven symlink generator
Git stores these blobs once (identical to existing history), so pack size is
effectively unchanged despite ~856MB of working-tree markdown.
Loading 73 doc versions concurrently exhausts the file-descriptor ulimit (EMFILE) on hosts with low limits. Preload graceful-fs + a fs.promises semaphore via NODE_OPTIONS in the build script so concurrent file ops stay well under the limit regardless of host ulimit. Adds graceful-fs as an explicit dependency.
- graceful-fs-preload: also wrap graceful-fs's own exports (fs-extra uses
require('graceful-fs') directly, bypassing core fs), with a concurrency
semaphore on read/stat/readdir across callback + promises APIs. This bounds
open FDs during parallel multi-version loading, fixing EMFILE.
- sanitize_sidebars.py: remove versioned-sidebar references to doc ids absent
from a snapshot (transient historical states where sidebars.js was ahead of
docs/), which Docusaurus otherwise rejects. Wired into generate_versions.sh.
Building all 73 versions needs >8GB heap and 30+ min, exceeding a default Vercel builder. Default to building the latest 20 released versions (+ main) via onlyIncludeVersions so deploys stay fast/light on standard hosting; all 73 snapshots remain committed. Set DOCS_VERSIONS_BUILD_LIMIT=all (or a number) on a larger builder to publish more. The /versions page notes when older releases are archived rather than built.
Webpack OOMs (>8GB heap) bundling many doc versions, even ~20. Switch to the Rust-based rspack bundler (via @docusaurus/faster experimental_faster) which uses a fraction of the memory, making the versioned build feasible on standard hosts.
fs-extra probes fs.realpath.native; the semaphore wrapper dropped it, triggering a 'is fs being monkey-patched?' warning. Copy original own-properties onto the wrapper.
Docusaurus re-renders every frozen version on every build, blowing Vercel's 45-min merge gate. Decouple: - Default build (Vercel, every PR + prod) now renders CURRENT docs only — fast. - DOCS_VERSIONS_BUILD_LIMIT=all builds the full archive; new GitHub Actions workflow build-docs-archive.yml does this off the merge path (no 45-min cap) and deploys to GitHub Pages, rebuilding only when versions change. - Custom manifest-driven version dropdown + /versions page link frozen versions to the archive via DOCS_ARCHIVE_URL. - url/baseUrl overridable; archive build sets noIndex + canonical to the live site so it doesn't compete in search. See versioning/README.md for setup.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The 73 version snapshots (48k files, ~12M lines) don't belong in the repo — they are derived artifacts. Untrack versioned_docs/, versioned_sidebars/, versions.json and gitignore them; the archive workflow now regenerates them from git history (fetch-depth: 0 + generate_versions.sh) before the full build. The committed manifest.json drives the version dropdown / Versions page, so the fast Vercel build (which has no snapshots on disk) still shows the version picker, linking frozen versions to the archive. Shrinks the PR diff to the versioning machinery.
Per project decision, version only STABLE releases from 1.83.x up (1.83.x line uses -stable tags -> 1.83.10-stable labels; 1.84.0+ are semver finals) plus the latest rc. That's ~23 versions, small enough to build directly on Vercel — so the GitHub Pages archive + current-only split are removed. - build_manifest.py: selects the stable set + latest rc, maps each to a commit. - prepare-snapshots.sh: npm prebuild hook regenerates snapshots from git history at build time (deepens history; degrades to current-only if unavailable), so the 48k snapshot files are never committed. - config: latest STABLE is the default at /docs/ (not the rc); rc + main get 'unreleased' banners; native version dropdown. - /versions: lists stable + rc sections from the manifest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Today the docs have no versioning, so a user on, say,
litellm==1.82.0can't tell what's changed between their version and the latest — a frequent source of confusion. This adds per-pip-release Docusaurus versioning, backfilled from the git history preserved through the docs migration.What's included
X.Y.Zpip release from 1.79.0 (2025-10-26) → 1.87.1, each reconstructed from the docs as they existed at that release's commit. (.dev/rc/postbuilds excluded — not prod install targets.)versioning/tooling (reproducible, idempotent):build_manifest.py(maps each release → docs commit by PyPI publish date),generate_versions.sh, plus helpers for image/sidebar consistency./versionspage, an "unreleased" banner onmainand "unmaintained" banner on old versions, and noindex + canonical-to-latest on archived versions so they don't dilute SEO/search.Build architecture (the important part)
Docusaurus re-renders every version on every build, but the snapshots are frozen. Rebuilding 73 versions on each PR blew past Vercel's 45-min limit (and OOM'd at 8 GB). So the build is split — the merge-gating build never touches the snapshots:
.github/workflows/build-docs-archive.yml)Controlled by
DOCS_VERSIONS_BUILD_LIMIT(currentdefault ·all·<N>). Memory was solved by enabling the rspack bundler; agraceful-fspreload bounds file-descriptor concurrency for the many-version archive build.⚙️ Setup required to go live (one-time)
build-docs-archive.ymlcan publish the archive.DOCS_ARCHIVE_URLto the archive base (e.g.https://berriai.github.io/litellm-docs) so the dropdown//versionslinks resolve. The Vercel build stayscurrent-only automatically — nothing else to change.docs.litellm.ai/docs/<version>/instead of cross-origin.Full details, the version→commit mapping caveat, and how to add future versions are in
versioning/README.md.Validation
Not included (deliberately deferred)
release_notes/andblog/remain global (unversioned), as intended.https://claude.ai/code/session_01WS4DCjMGq4TdxuecNEc9Jc
Generated by Claude Code