Repository navigation
Conversation
apache/datafusion#27337 removed `datafusion-cli --spark`, so there is no longer a way to try the Spark-compatible functions interactively. Add a `datafusion-spark-cli` binary that uses the `datafusion-cli` library for the shell, with the Spark SQL dialect and functions always enabled and ANSI mode on by default, as in Spark 4. The crate is not published, and is not a default workspace member, so a plain `cargo build` or `cargo test` does not build its dependencies. A new workflow builds it and runs a smoke test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kosiew
left a comment
There was a problem hiding this comment.
@andygrove, thanks for this. Reusing DataFusion's CLI while configuring Spark-specific behavior is the right approach.
Requesting changes because relative script paths can resolve incorrectly or panic when combined with --data-path. The remaining suggestions improve CI coverage and documentation.
| ); | ||
| } | ||
|
|
||
| if let Some(path) = &args.data_path { |
There was a problem hiding this comment.
With -p, relative -f and -r paths can execute the wrong file or panic. Please absolutize script paths before changing directories, preserving /dev/stdin, and use exec_from_lines to handle file-open errors. Test both flags with conflicting filenames.
There was a problem hiding this comment.
Good catch, thanks. Fixed in f0cf6de. With -p, the -f and -r paths are now made absolute before the directory change, and the stdin names are left as they are. The files are opened here and run with exec_from_lines, so a file that can't be opened is an error rather than a panic. I also found that -f /dev/stdin was rejected at argument parsing because a pipe isn't a regular file, so parse_valid_file now accepts the stdin names. CI runs both flags with a q.sql in the working directory and a different q.sql in the data directory. The step fails without the fix.
| run: | | ||
| set -euo pipefail | ||
| cli=target/debug/datafusion-spark-cli | ||
| $cli -q --format csv -c "SELECT sha2('Spark', 256) AS h" \ |
There was a problem hiding this comment.
Could we add a stdin REPL smoke test using CAST(1 AS LONG) and assert the result row? Current queries also pass with the generic dialect, so they do not verify Spark parsing.
There was a problem hiding this comment.
Added in f0cf6de. CI pipes SELECT CAST(1 AS LONG) AS l; into the REPL and checks for the row 1.
| cli=target/debug/datafusion-spark-cli | ||
| $cli -q --format csv -c "SELECT sha2('Spark', 256) AS h" \ | ||
| | grep -x '529bc3b07127ecb7e53a4dcf1991d9152c24537d919178022b2c42657f79a26b' | ||
| if $cli -q -c "SELECT abs(cast(-2147483648 AS int))"; then |
There was a problem hiding this comment.
Please also assert [ARITHMETIC_OVERFLOW] in the captured output. A startup failure or panic could otherwise satisfy the nonzero-exit check.
There was a problem hiding this comment.
Done in f0cf6de. The step captures the output and checks it contains [ARITHMETIC_OVERFLOW].
| The option is disabled by default because Spark-compatible functions can | ||
| override DataFusion functions with the same name. | ||
| The Spark functions override DataFusion functions with the same name. ANSI mode | ||
| is on by default, as in Spark 4. Pass `--ansi false` for Spark's legacy behavior. |
There was a problem hiding this comment.
Please document cargo run -p datafusion-spark-cli -- --ansi false here and in the SQLLogicTest README. The -- separator is required to pass --ansi to the CLI.
There was a problem hiding this comment.
Done in f0cf6de. Both READMEs now show cargo run -p datafusion-spark-cli -- --ansi false and explain the --.
With -p the CLI changed directory before opening the -f and -r files, so a relative path ran the file of that name in the data directory, or panicked if there was none. Make those paths absolute first, open the files ourselves so a failure is an error rather than a panic, and accept -f /dev/stdin. CI now also runs the REPL from stdin with a Spark-only type, checks the ARITHMETIC_OVERFLOW message, and runs -f and -r with a data path. The READMEs show that --ansi false goes after -- with cargo run.
Which issue does this PR close?
Part of #56. It covers the
datafusion-cli --sparkreferences in the "Update the imported docs" item, since those READMEs now point to the new CLI.Rationale for this change
apache/datafusion#27337 removed
datafusion-cli --spark, which enabled the Spark SQL dialect and registered the Spark-compatible functions. Since then there is no way to try the functions interactively, andcrates/spark/README.mdandcrates/sqllogictest/test_files/README.mdstill tell people to rundatafusion-cli --spark.What changes are included in this PR?
crates/cli, a newdatafusion-spark-clibinary. It uses thedatafusion-clilibrary for the REPL, for running files and commands, for output formats, and for querying files by path or URL (SELECT * FROM 'data.parquet'). Itsmain.rsis about 200 lines, followingdatafusion-cli's.spark.sql.ansi.enabled = true).--ansi falsegives Spark's legacy behavior. The startup banner says which mode is on.datafusion-cli:-c,-f,-r,-p,-b,--format,-q,--maxrowsand--color. The memory and disk tuning options are left out for now.publish = false.datafusion-clibrings in the object store and AWS dependencies, so a plaincargo buildorcargo testat the root skips this crate. The clippy job uses--workspace, so it still lints the crate..github/workflows/cli.yml. It builds the CLI and runs a smoke test:sha2returns Spark's result.abs(cast(-2147483648 AS int))fails by default, because ANSI mode is on.-2147483648with--ansi false.crates/spark/README.md,test_files/README.md, the rootREADME.mdandAGENTS.mddescribe the new CLI.datafusion-clion DataFusion'sbranch-56still depends on that branch's owndatafusion-spark, so the build compiles that copy too. Only this repository's functions are registered. The extra copy goes away with DataFusion 57, wheredatafusion-clino longer depends ondatafusion-spark.Are these changes tested?
--helpand the banner.cargo clippy --all-targets --all-features --workspace -- -D warningsis clean.cargo test -p datafusion-sparkandcargo test -p datafusion-spark-sqllogicteststill pass.actionlintreports no problems with the new workflow.Are there any user-facing changes?
There is a new, unpublished
datafusion-spark-clibinary. You run it withcargo run -p datafusion-spark-cli.This PR was written by an LLM (Claude) and reviewed before opening.
🤖 Generated with Claude Code