Skip to content

docs: add a versioning and API compatibility policy - #319

Merged
mbutrovich merged 1 commit into
apache:mainfrom
andygrove:docs-versioning-policy
Nov 16, 2026
Merged

mbutrovich merged 1 commit into
apache:mainfrom
andygrove:docs-versioning-policy

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #56. It replaces that issue's plan to make 57.0.0 the first release from this repository.

Rationale for this change

New Spark-compatible functions and fixes land here between DataFusion major releases. Tying datafusion-spark releases to DataFusion's would hold them back, so this PR proposes releasing 56.x from main as often as needed, until DataFusion 57.0.0 is released. That only works if main doesn't break the public API in the meantime, so the policy spells out what counts as a breaking change.

What changes are included in this PR?

  • dev/release/README.md: expands the Versioning section.
    • The major version matches DataFusion's. Minor releases (such as 56.1.0) add functions, features and fixes. Patch releases (such as 56.0.1) contain only fixes.
    • main stays on DataFusion 56 and releases 56.x until DataFusion 57.0.0 is released. Then it moves to DataFusion 57, and its next release is 57.0.0.
    • DataFusion 56.0.0 publishes datafusion-spark 56.0.0 from branch-56 in apache/datafusion. Later 56.x releases come from this repository, so a DataFusion 56.x patch release should not publish datafusion-spark again.
    • What a 56.x release must not break, what is allowed, and how to avoid a break with #[deprecated]. A change to a function's results that makes it match Spark counts as a bug fix.
    • Checking a release against the previous one with cargo-semver-checks.
  • CONTRIBUTING.md: a short "API Compatibility" section for contributors that links to the policy.

Are these changes tested?

Documentation only. To check that the policy can be applied today, I ran cargo semver-checks -p datafusion-spark --all-features against the crate at apache/datafusion's 56.0.0-rc1 tag. Results:

Compared Result
Current main No breaking changes (229 checks pass)
The open cast PRs (#89, #304, #317) No breaking changes
A test rename of a public function Reported as a major-version change, so the check does detect breaks

Since rc1, main has only added public items, such as dayname and date_from_unix_date.

Are there any user-facing changes?

No code changes.

The policy needs agreement with apache/datafusion: its branch-56 release process still runs cargo publish for datafusion-spark, so DataFusion 56.x patch releases would need to stop publishing it after 56.0.0.

This PR was written by an LLM (Claude) and reviewed before opening.

🤖 Generated with Claude Code

Keep datafusion-spark on 56.x until DataFusion 57.0.0 is released, and
release 56.x minor and patch versions from main in the meantime, so new
functions and fixes ship without waiting for DataFusion. Spell out what
counts as a breaking API change, which has to wait for the next
DataFusion major version, and how to check for one with
cargo-semver-checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove

Copy link
Copy Markdown
Member Author

@comphead @mbutrovich @shehabgamin moving this code to a new repo means that we can actually release datafusion-spark more frequently than core DataFusion now as we add new expressions and fix bugs. Would you mind reviewing this PR?

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this versioning policy makes sense. Let's see if we run into hiccups. Thanks @andygrove!

@mbutrovich
mbutrovich merged commit 627f9c3 into apache:main Nov 16, 2026
4 checks passed
@comphead

Copy link
Copy Markdown
Contributor

Two follow-ups on the API compatibility section:

  1. #[non_exhaustive] wording. The attribute applies to structs, enums and enum variants, not to fields, so "new #[non_exhaustive] fields" can't be done. The rule is also stricter than needed: adding a field only breaks a struct whose fields are all public. Suggested wording: "add a public field to a struct whose fields are all public, or a variant to an enum, unless the type is #[non_exhaustive]". Adding #[non_exhaustive] to an existing public type is itself a breaking change, and the section could say so.

  2. The SQL-facing API isn't covered. Most users reach the crate through register_all and all_default_*_functions, not through Rust items. cargo-semver-checks can't see changes such as dropping a function from those lists, renaming or removing a SQL name or alias, or changing a return type. A bullet listing these as breaking would help. So would a note that cargo-semver-checks doesn't cover them. It would also help to say whether a return-type change that makes a function match Spark counts as a bug fix, like a change in results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants