Skip to content

docs(spark): document hash, bitwise, bitmap and conditional functions with #[user_doc] - #107

Merged
andygrove merged 1 commit into
apache:mainfrom
andygrove:docs-user-doc-hash-bitwise
Nov 14, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
andygrove:docs-user-doc-hash-bitwise

Conversation

@andygrove

Copy link
Copy Markdown
Member

Part of #55

Rationale

#58 added #[user_doc] and documented sha2 and substring as examples. This PR applies the same approach to the remaining hash, bitwise, bitmap, equal_null and if functions, so they appear in generated function references.

Changes

  • Add #[user_doc] and documentation() to:
    • hash: sha1 (alias sha), crc32
    • bitwise: bit_get (alias getbit), bit_count, bitwise_not, shiftleft, shiftright, shiftrightunsigned
    • bitmap: bitmap_count, bitmap_bucket_number, bitmap_bit_position
    • misc: equal_null
    • conditional: if
  • Each description covers the Spark edge cases the tests check, for example:
    • bit_count sign-extends to 64 bits.
    • bit_get's position limit depends on the input type. Integer literals are BIGINT here and INT in Spark, so the limit for an uncast literal differs.
    • Shift amounts are taken modulo the bit width.
    • equal_null treats NaN as equal to NaN and 0.0 as equal to -0.0.
    • if only raises errors from the branch it takes.
  • shiftleft, shiftright and shiftrightunsigned share one struct, SparkBitShift. A single #[user_doc] on it would give all three the same syntax and examples. Instead, each function's docs sit on a private marker type, and documentation() picks one based on the shift direction.
  • Cut the structs' rustdoc down to a one-line summary and a link to the Spark docs. Spark SQL has no bitwise_not function (it uses ~), so that link points to the PySpark docs.
  • I didn't touch sha2, which is already documented, or xxhash64, which feat(spark): add Spark-compatible hash (Murmur3) function #68 changes.

Testing

  • cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, cargo test -p datafusion-spark and cargo test -p datafusion-spark-sqllogictest pass.
  • I ran every SQL example in the docs (36 queries) through the sqllogictest runner, and they all match.
  • Everything the docs describe is covered by the existing SLT or unit tests for these functions, or is visible in the code.

User-facing changes

documentation() now returns docs for these functions. Behavior does not change.

🤖 Generated with Claude Code

… with #[user_doc]

Add #[user_doc] documentation and documentation() to:

- hash: sha1, crc32
- bitwise: bit_get, bit_count, bitwise_not, shiftleft, shiftright,
  shiftrightunsigned
- bitmap: bitmap_count, bitmap_bucket_number, bitmap_bit_position
- misc: equal_null
- conditional: if

The three shift functions share one struct, so each gets its own
documentation via a private marker type, selected by shift direction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@coderfender coderfender left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andygrove
andygrove merged commit aaa37ca into apache:main Nov 14, 2026
4 checks passed
andygrove added a commit that referenced this pull request Nov 19, 2026
Part of #55

## Rationale

The recent `#[user_doc]` PRs (#58, #103–#107) attach user documentation
to most functions, but nothing in the repo renders it. `cargo doc`
doesn't show it either, since `#[user_doc]` builds a runtime
`Documentation` value rather than rustdoc, and those PRs cut the rustdoc
down to a one-line summary. So the docs can't be read anywhere yet.

## Changes

- Add `crates/docgen` (`datafusion-spark-docgen`, `publish = false`)
with a `print_functions_docs` binary. It is based on DataFusion's
`print_functions_docs`. It writes Markdown for all scalar, aggregate and
window functions returned by `all_default_*_functions()`, with
arguments, examples, alternative syntax and aliases.
- Output goes to `docs/functions`, with one page per function type and
doc section (for example `scalar/math.md`) and a `README.md` index. The
index links each section with its function count, then lists every
function and alias alphabetically. A single page was about 2,500 lines
already and would reach roughly 10,000 with full Spark coverage.
- Sections follow DataFusion's order. Sections DataFusion doesn't define
(`JSON Functions`, `URL Functions`) come after, sorted by label.
- Unlike DataFusion, it doesn't fail on undocumented functions, because
#55 is still in progress. It lists them on stderr instead (10 right now:
`array_contains`, `map_from_arrays`, `map_from_entries`, `mod`,
`parse_url`, `pmod`, `spark_cast`, `str_to_map`, `try_parse_url`,
`xxhash64`).
- Add `dev/update_function_docs.sh`, which generates into a temporary
directory and replaces `docs/functions` only on success, and commit the
generated pages.
- Add a `function docs up to date` job to `lint.yml` that regenerates
the pages and fails if `git status --porcelain -- docs/functions`
reports any change, including deleted or untracked files.
- Document the workflow in `CONTRIBUTING.md` and `AGENTS.md`, and link
the reference from the READMEs and crate docs.

The published `datafusion-spark` crate is unchanged apart from a doc
comment link.

## Testing

- `cargo fmt --all -- --check`, `cargo clippy --all-targets
--all-features --workspace -- -D warnings`, `cargo test -p
datafusion-spark` and `cargo test -p datafusion-spark-sqllogictest`
pass.
- Ran `dev/update_function_docs.sh` and checked the rendered pages. The
split pages contain the same functions and aliases as the earlier single
page, and a second run leaves the tree clean.
- Made `cargo` fail during generation and checked that the existing
pages were left untouched.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants