Skip to content

feat: add datafusion-spark-cli, a SQL shell with the Spark functions - #326

Open
andygrove wants to merge 4 commits into
apache:mainfrom
andygrove:feat-spark-cli
Open

andygrove wants to merge 4 commits into
apache:mainfrom
andygrove:feat-spark-cli

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #56. It covers the datafusion-cli --spark references in the "Update the imported docs" item, since those READMEs now point to the new CLI.

Rationale for this change

apache/datafusion#27337 removed datafusion-cli --spark, which enabled the Spark SQL dialect and registered the Spark-compatible functions. Since then there is no way to try the functions interactively, and crates/spark/README.md and crates/sqllogictest/test_files/README.md still tell people to run datafusion-cli --spark.

What changes are included in this PR?

  • crates/cli, a new datafusion-spark-cli binary. It uses the datafusion-cli library for the REPL, for running files and commands, for output formats, and for querying files by path or URL (SELECT * FROM 'data.parquet'). Its main.rs is about 200 lines, following datafusion-cli's.
    • Always Spark: the Spark SQL dialect and the Spark functions are always enabled. The Spark functions override DataFusion's functions of the same name.
    • ANSI mode on by default, as in Spark 4 (spark.sql.ansi.enabled = true). --ansi false gives Spark's legacy behavior. The startup banner says which mode is on.
    • Options kept from datafusion-cli: -c, -f, -r, -p, -b, --format, -q, --maxrows and --color. The memory and disk tuning options are left out for now.
  • Not published: publish = false.
  • Not a default workspace member: datafusion-cli brings in the object store and AWS dependencies, so a plain cargo build or cargo test at the root skips this crate. The clippy job uses --workspace, so it still lints the crate.
  • New workflow, .github/workflows/cli.yml. It builds the CLI and runs a smoke test:
    • sha2 returns Spark's result.
    • abs(cast(-2147483648 AS int)) fails by default, because ANSI mode is on.
    • The same query returns -2147483648 with --ansi false.
  • Docs: crates/spark/README.md, test_files/README.md, the root README.md and AGENTS.md describe the new CLI.

datafusion-cli on DataFusion's branch-56 still depends on that branch's own datafusion-spark, so the build compiles that copy too. Only this repository's functions are registered. The extra copy goes away with DataFusion 57, where datafusion-cli no longer depends on datafusion-spark.

Are these changes tested?

  • The smoke test's three queries pass locally.
  • I also tried the REPL reading from stdin, a backquoted identifier (Spark dialect), --help and the banner.
  • cargo clippy --all-targets --all-features --workspace -- -D warnings is clean.
  • cargo test -p datafusion-spark and cargo test -p datafusion-spark-sqllogictest still pass.
  • actionlint reports no problems with the new workflow.

Are there any user-facing changes?

There is a new, unpublished datafusion-spark-cli binary. You run it with cargo run -p datafusion-spark-cli.

This PR was written by an LLM (Claude) and reviewed before opening.

🤖 Generated with Claude Code

apache/datafusion#27337 removed `datafusion-cli --spark`, so there is no
longer a way to try the Spark-compatible functions interactively. Add a
`datafusion-spark-cli` binary that uses the `datafusion-cli` library for
the shell, with the Spark SQL dialect and functions always enabled and
ANSI mode on by default, as in Spark 4.

The crate is not published, and is not a default workspace member, so a
plain `cargo build` or `cargo test` does not build its dependencies. A
new workflow builds it and runs a smoke test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove, thanks for this. Reusing DataFusion's CLI while configuring Spark-specific behavior is the right approach.

Requesting changes because relative script paths can resolve incorrectly or panic when combined with --data-path. The remaining suggestions improve CI coverage and documentation.

Comment thread crates/cli/src/main.rs
);
}

if let Some(path) = &args.data_path {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With -p, relative -f and -r paths can execute the wrong file or panic. Please absolutize script paths before changing directories, preserving /dev/stdin, and use exec_from_lines to handle file-open errors. Test both flags with conflicting filenames.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Fixed in f0cf6de. With -p, the -f and -r paths are now made absolute before the directory change, and the stdin names are left as they are. The files are opened here and run with exec_from_lines, so a file that can't be opened is an error rather than a panic. I also found that -f /dev/stdin was rejected at argument parsing because a pipe isn't a regular file, so parse_valid_file now accepts the stdin names. CI runs both flags with a q.sql in the working directory and a different q.sql in the data directory. The step fails without the fix.

Comment thread .github/workflows/cli.yml
run: |
set -euo pipefail
cli=target/debug/datafusion-spark-cli
$cli -q --format csv -c "SELECT sha2('Spark', 256) AS h" \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a stdin REPL smoke test using CAST(1 AS LONG) and assert the result row? Current queries also pass with the generic dialect, so they do not verify Spark parsing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in f0cf6de. CI pipes SELECT CAST(1 AS LONG) AS l; into the REPL and checks for the row 1.

Comment thread .github/workflows/cli.yml Outdated
cli=target/debug/datafusion-spark-cli
$cli -q --format csv -c "SELECT sha2('Spark', 256) AS h" \
| grep -x '529bc3b07127ecb7e53a4dcf1991d9152c24537d919178022b2c42657f79a26b'
if $cli -q -c "SELECT abs(cast(-2147483648 AS int))"; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also assert [ARITHMETIC_OVERFLOW] in the captured output. A startup failure or panic could otherwise satisfy the nonzero-exit check.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f0cf6de. The step captures the output and checks it contains [ARITHMETIC_OVERFLOW].

Comment thread crates/spark/README.md Outdated
The option is disabled by default because Spark-compatible functions can
override DataFusion functions with the same name.
The Spark functions override DataFusion functions with the same name. ANSI mode
is on by default, as in Spark 4. Pass `--ansi false` for Spark's legacy behavior.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please document cargo run -p datafusion-spark-cli -- --ansi false here and in the SQLLogicTest README. The -- separator is required to pass --ansi to the CLI.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f0cf6de. Both READMEs now show cargo run -p datafusion-spark-cli -- --ansi false and explain the --.

With -p the CLI changed directory before opening the -f and -r files, so a
relative path ran the file of that name in the data directory, or panicked if
there was none. Make those paths absolute first, open the files ourselves so
a failure is an error rather than a panic, and accept -f /dev/stdin.

CI now also runs the REPL from stdin with a Spark-only type, checks the
ARITHMETIC_OVERFLOW message, and runs -f and -r with a data path. The READMEs
show that --ansi false goes after -- with cargo run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants