Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .github/workflows/cli.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements

name: CLI

on:
push:
branches: [main]
pull_request:

permissions:
contents: read

jobs:
smoke-test:
name: datafusion-spark-cli smoke test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: rustup toolchain install stable --profile minimal --no-self-update && rustup default stable
- uses: Swatinem/rust-cache@v2
- run: cargo build -p datafusion-spark-cli
# A Spark function, and ANSI mode on by default and off with `--ansi false`.
- name: Run queries
run: |
set -euo pipefail
cli=target/debug/datafusion-spark-cli
$cli -q --format csv -c "SELECT sha2('Spark', 256) AS h" \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a stdin REPL smoke test using CAST(1 AS LONG) and assert the result row? Current queries also pass with the generic dialect, so they do not verify Spark parsing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in f0cf6de. CI pipes SELECT CAST(1 AS LONG) AS l; into the REPL and checks for the row 1.

| grep -x '529bc3b07127ecb7e53a4dcf1991d9152c24537d919178022b2c42657f79a26b'
if out=$($cli -q -c "SELECT abs(cast(-2147483648 AS int))" 2>&1); then
echo "expected an ARITHMETIC_OVERFLOW error with ANSI mode on" && exit 1
fi
grep -F '[ARITHMETIC_OVERFLOW]' <<< "$out"
$cli -q --ansi false --format csv -c "SELECT abs(cast(-2147483648 AS int)) AS a" \
| grep -x -- '-2147483648'
# SQL read from stdin by the REPL. `LONG` is a type in the Spark dialect only.
- name: Run the REPL from stdin
run: |
set -euo pipefail
echo "SELECT CAST(1 AS LONG) AS l;" \
| target/debug/datafusion-spark-cli -q --format csv | grep -x '1'
# `-f` and `-r` paths are relative to the working directory, not to `-p`.
- name: Run files with a data path
run: |
set -euo pipefail
cli=$PWD/target/debug/datafusion-spark-cli
dir=$(mktemp -d) && cd "$dir" && mkdir data
echo "SELECT 'cwd' AS s;" > q.sql
echo "SELECT 'data' AS s;" > data/q.sql
$cli -q --format csv -p data -f q.sql | grep -x 'cwd'
echo "" | $cli -q --format csv -p data -r q.sql | grep -x 'cwd'
echo "SELECT 'stdin' AS s;" | $cli -q --format csv -p data -f /dev/stdin | grep -x 'stdin'
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ DataFusion expressions that emulate Apache Spark's behavior.

- `crates/spark` – the `datafusion-spark` crate
- `crates/sqllogictest` – `.slt` tests for the Spark functions
- `crates/cli` – `datafusion-spark-cli`, a SQL shell for trying the functions (not a default
workspace member, so build it with `-p datafusion-spark-cli`)

Before opening a PR, run `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`,
`cargo test -p datafusion-spark` and `cargo test -p datafusion-spark-sqllogictest`.
Expand Down
Loading
Loading