Skip to content

docs(spark): generate function reference from #[user_doc] - #115

Merged
andygrove merged 8 commits into
apache:mainfrom
andygrove:docs-function-reference-generator
Nov 19, 2026
Merged

andygrove merged 8 commits into
apache:mainfrom
andygrove:docs-function-reference-generator

Conversation

@andygrove

@andygrove andygrove commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Part of #55

Rationale

The recent #[user_doc] PRs (#58, #103–#107) attach user documentation to most functions, but nothing in the repo renders it. cargo doc doesn't show it either, since #[user_doc] builds a runtime Documentation value rather than rustdoc, and those PRs cut the rustdoc down to a one-line summary. So the docs can't be read anywhere yet.

Changes

  • Add crates/docgen (datafusion-spark-docgen, publish = false) with a print_functions_docs binary. It is based on DataFusion's print_functions_docs. It writes Markdown for all scalar, aggregate and window functions returned by all_default_*_functions(), with arguments, examples, alternative syntax and aliases.
    • Output goes to docs/functions, with one page per function type and doc section (for example scalar/math.md) and a README.md index. The index links each section with its function count, then lists every function and alias alphabetically. A single page was about 2,500 lines already and would reach roughly 10,000 with full Spark coverage.
    • Sections follow DataFusion's order. Sections DataFusion doesn't define (JSON Functions, URL Functions) come after, sorted by label.
    • Unlike DataFusion, it doesn't fail on undocumented functions, because Epic: Document Spark-specific behavior for every expression #55 is still in progress. It lists them on stderr instead (10 right now: array_contains, map_from_arrays, map_from_entries, mod, parse_url, pmod, spark_cast, str_to_map, try_parse_url, xxhash64).
  • Add dev/update_function_docs.sh, which generates into a temporary directory and replaces docs/functions only on success, and commit the generated pages.
  • Add a function docs up to date job to lint.yml that regenerates the pages and fails if git status --porcelain -- docs/functions reports any change, including deleted or untracked files.
  • Document the workflow in CONTRIBUTING.md and AGENTS.md, and link the reference from the READMEs and crate docs.

The published datafusion-spark crate is unchanged apart from a doc comment link.

Testing

  • cargo fmt --all -- --check, cargo clippy --all-targets --all-features --workspace -- -D warnings, cargo test -p datafusion-spark and cargo test -p datafusion-spark-sqllogictest pass.
  • Ran dev/update_function_docs.sh and checked the rendered pages. The split pages contain the same functions and aliases as the earlier single page, and a second run leaves the tree clean.
  • Made cargo fail during generation and checked that the existing pages were left untouched.

andygrove and others added 3 commits November 14, 2026 10:30
Add a datafusion-spark-docgen crate with a print_functions_docs binary,
dev/update_function_docs.sh to write docs/functions.md, the generated
file, and a CI check that keeps it up to date.

Part of apache#55

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ce-generator

# Conflicts:
#	.github/workflows/lint.yml

@kosiew kosiew left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove, thanks for this change. Generating the reference from existing UDF metadata is appropriate and keeps documentation aligned with the registered functions.

Approving.
The remaining suggestions concern CI coverage, regeneration safety, and wording accuracy. None are blocking.

Comment thread .github/workflows/lint.yml Outdated
- run: dev/update_function_docs.sh
- name: Check docs/functions.md is up to date
run: |
if ! git diff --exit-code -- docs/functions.md; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use git status --porcelain -- docs/functions.md instead? git diff --exit-code misses untracked files, so CI passes if the reference is deleted and regenerated.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Switched to git status --porcelain -- docs/functions.md, which also catches deleted and untracked files, and the step now prints the status and diff on failure. Fixed in 7233f36.

Comment thread dev/update_function_docs.sh Outdated
differs from the DataFusion built-in of the same name, if there is one.
HEADER
cargo run --quiet -p datafusion-spark-docgen --bin print_functions_docs
} > "${TARGET_FILE}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A failed cargo run leaves the reference truncated to its header. Could we generate into a temporary file, replace the target only on success, and clean up with a trap?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7233f36. The script generates into a mktemp file and moves it over docs/functions.md only on success. A trap removes the temp file on any exit.

Comment thread crates/spark/README.md Outdated

Not every Spark function is implemented yet. Progress is tracked in the
[`datafusion-spark` epic]; see the [API documentation] for the functions available today.
[`datafusion-spark` epic]. See the [function reference] for the functions available today, with

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reference excludes registered functions without documentation(). Could we say "documented functions" here and in README.md:52 and crates/spark/src/lib.rs:49 to avoid overstating coverage?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, 10 registered functions still lack documentation(). Changed to "documented functions" in crates/spark/README.md, README.md and crates/spark/src/lib.rs in 7233f36.

- Generate into a temporary file so a failed build leaves docs/functions.md intact.
- Use git status --porcelain in CI so a deleted or untracked reference is caught.
- Say "documented functions", since functions without documentation() are omitted.
docs/functions.md had grown to about 2,500 lines and would reach roughly
10,000 with full Spark coverage. The generator now writes
docs/functions/README.md, an index of categories and of every function, and
one page per function type and doc section, such as scalar/math.md.

Links between related functions resolve across pages.
@andygrove

Copy link
Copy Markdown
Member Author

@kosiew thanks for the review. Since your approval I've split the function reference into one page per category, because the single page was already about 2,500 lines and would reach roughly 10,000 with full Spark coverage. docs/functions/README.md is now an index of categories and of every function, and each category has its own page, for example scalar/math.md. The script and CI check cover the whole directory. The change is in faa160c. Could you take another look?

@andygrove
andygrove merged commit c79cd9f into apache:main Nov 19, 2026
9 checks passed
@andygrove
andygrove deleted the docs-function-reference-generator branch November 19, 2026 15:25
andygrove added a commit that referenced this pull request Nov 19, 2026
#378)

#115 added a CI check that `docs/functions` matches the `#[user_doc]`
attributes. It merged after #329 and #357, which changed those
attributes, so the Lint workflow fails on main. This reruns
`dev/update_function_docs.sh`. It only changes generated docs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants