Repository navigation
Open
Labels
EPICA larger project, actively underway, with sub tasksA larger project, actively underway, with sub tasks
Description
Tracks the remaining work to finish moving datafusion-spark here from apache/datafusion, following #87.
Build and CI
- Add a workspace
Cargo.toml(chore: add top-level workspace Cargo.toml #45) - Agree on the versioning policy (docs: add a versioning and API compatibility policy #319).
datafusion-sparkstays on 56.x until DataFusion 57.0.0 is released. apache/datafusion no longer publishesdatafusion-sparkafter 56.0.0, since chore: remove thedatafusion-sparkcrate (moving to apache/datafusion-spark) datafusion#27337 removed the crate - Pin
mainto the DataFusion56.0.0-rc2tag instead ofbranch-56(build: depend on the DataFusion 56.0.0-rc2 tag instead of branch-56 #370), then move to the DataFusion 56.0.0 release on crates.io once it is published - Merge a first workflow
-
cargo fmtandcargo clippy(ci: add cargo fmt and clippy workflow #53) - Run the sqllogictest files (test: run sqllogictest files in CI #51)
- License header check (ci: add Apache RAT license header check #50, with Apache RAT). apache/datafusion checks the same SPDX headers with HawkEye
- Run
cargo test -p datafusion-spark(unit tests and doctests) (ci: run the datafusion-spark unit tests and doctests #318) - Check that the crate builds with default features, with no default features, and with the
corefeature, like thecargo check datafusion-spark featuresjob in apache/datafusion (ci: check that datafusion-spark builds with each feature combination #320) - Build the docs with
RUSTDOCFLAGS="-D warnings"(ci: build the rustdoc docs with warnings denied #321) - Add the ASF action allowlist check, as in apache/datafusion-iceberg (ci: check that workflows only use ASF-allowlisted actions #322)
- Require the CI checks in
.asf.yaml(ci: require the CI checks before merging to main #371)
Deferred: a Dependabot configuration (#323).
Documentation and repository files
- README,
LICENSE.txtandNOTICE.txt(docs: add README, LICENSE.txt and NOTICE.txt #49) - The
audit-datafusion-spark-expressionagent skill andAGENTS.md(docs: add audit-datafusion-spark-expression agent skill and AGENTS.md #52) - Add the AI policy's rule that contributions from fully autonomous agents are not accepted to
AGENTS.mdorCONTRIBUTING.md - Pull request template, and issue templates with a field for Spark versions (docs: add pull request and issue templates #325)
-
CONTRIBUTING.md(docs: improve READMEs and add CONTRIBUTING.md #60) - Replace
datafusion-cli --spark, which chore: remove thedatafusion-sparkcrate (moving to apache/datafusion-spark) datafusion#27337 removed, with adatafusion-spark-clibinary in this repository, and pointcrates/spark/README.mdandcrates/sqllogictest/test_files/README.mdto it (feat: add datafusion-spark-cli, a SQL shell with the Spark functions #326) - Update the rest of the imported docs.
crates/spark/README.mdstill describes the crate as a submodule of DataFusion and links to a sqllogictest README that has moved.crates/sqllogictest/test_files/README.mdlinks to [Ballista] Fix Ballista IT datafusion#5 by its old URL (apache/datafusion#15914). Both READMEs also change in test: run sqllogictests with and without constant folding #316 and feat: add datafusion-spark-cli, a SQL shell with the Spark functions #326, so this is easier after those merge
Moving work from apache/datafusion
- Post instructions in [DISCUSSION] How should we test datafusion-spark against Spark? #87 for porting open pull requests
- Port the PySpark runner for the sqllogictest files from chore: add a manual PySpark runner for the Spark sqllogictest files datafusion#27165 (chore: add a manual PySpark runner for the Spark sqllogictest files #54)
- Port or close the open
sparkpull requests in apache/datafusion (14 open as of 2026-11-16) - Decide whether to transfer Decimal division compatibility mode with spark datafusion#7301, Deduplicate Spark function code with native/default datafusion function code datafusion#17964, [Refactor] Unify Spark Avg and Built-in Avg Group-Accumulator State Handling datafusion#22638, [EPIC] Support ANSI mode datafusion#23929, array_agg accumulators do per-element work when merging partial states datafusion#25118 and DataFrame API: serialize columns to a JSON string column datafusion#26185 (DataFrame API: serialize columns to a JSON string column datafusion#26185 is now closed)
- Merge chore: remove the
datafusion-sparkcrate (moving to apache/datafusion-spark) datafusion#27337, which removes the crate from apache/datafusion - Bring over any Spark changes that landed in apache/datafusion before chore: remove the
datafusion-sparkcrate (moving to apache/datafusion-spark) datafusion#27337
Releases
- Add release scripts under
dev/release/and changelog generation (docs: add release process documentation and release scripts #114) - Release
datafusion-spark56.1.0, the first release from this repository, after DataFusion 56.0.0 - Release 57.0.0 after DataFusion 57.0.0
Until DataFusion 56.0.0 is released, fixes for datafusion-spark 56.0.0 need to be backported to branch-56 in apache/datafusion.
Community
- Announce the move on dev@datafusion.apache.org
- Decide whether to create a Slack channel
- Agree who leads Spark CI, reviews and Spark version support
Testing against Spark
- Run the sqllogictest files against Apache Spark in CI (ci: run the PySpark sqllogictest runner on Spark #369)
The checklist was updated by an LLM (Claude) from the state of main and the open pull requests on 2026-11-18, and reviewed before posting.
Metadata
Metadata
Assignees
Labels
EPICA larger project, actively underway, with sub tasksA larger project, actively underway, with sub tasks
Activity
ci: run the datafusion-spark unit tests and doctests (#318)
ci: check that datafusion-spark builds with each feature combination (#…
ci: build the rustdoc docs with warnings denied (#321)
ci: check that workflows only use ASF-allowlisted actions (#322)
docs: add pull request and issue templates (#325)
docs: add a versioning and API compatibility policy (#319)