Skip to content

test: run sqllogictests with and without constant folding - #316

Merged
andygrove merged 1 commit into
apache:mainfrom
andygrove:test-slt-eval-modes
Nov 19, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
andygrove:test-slt-eval-modes

Conversation

@andygrove

@andygrove andygrove commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

None. It adds the test coverage that found #305–#315.

Rationale for this change

A Spark function can run in two ways. When every argument is a literal, the simplify_expressions optimizer rule constant-folds the call while planning. That calls ScalarUDFImpl::simplify (if the function has one) and evaluates the result on a one-row batch. Otherwise the call runs through invoke_with_args during execution. That is also the only path for engines such as Comet that build physical expressions directly.

Most .slt queries are literal-only, such as SELECT f(1), so today they only test the first path. invoke_with_args must always work, and simplify is an optional optimization. This PR runs the tests through both paths.

What changes are included in this PR?

  • The runner runs every .slt file twice:
    • simplify: the default optimizer, as before.
    • evaluate: the same optimizer rules with simplify_expressions replaced by a rule that rewrites only DataFusion's rewrite-only functions (arrow_cast, arrow_try_cast, cast_to_type, try_cast_to_type, nvl2). The tests use those to build typed inputs, and their invoke_with_args always errors. Every other call, including literal-only calls of the Spark functions, runs through invoke_with_args.
  • Each run gets its mode as an .slt label, so a record can be marked skipif evaluate. (skipif rather than onlyif simplify, because the PySpark runner skips onlyif records that are not for spark.)
  • --mode=simplify or --mode=evaluate runs a single mode. Results print as PASS [evaluate] path.
  • A panic in one file is reported as that file failing, instead of aborting the whole run.
  • Each record that cannot run in evaluate mode yet is marked skipif evaluate, with a note at the top of the file linking its issue.
  • --complete runs only in simplify mode, so each file is rewritten once.
  • test_files/README.md documents the two modes.

What evaluate mode found

Eight functions have no working invoke_with_args: they error unless simplify rewrites them first. Their records are marked skipif evaluate.

Function Issue
if #305
unix_date #306
date_diff #307
date_trunc #309
time_trunc #310
trunc #311
add_months #312
base64 #313

Two bugs in the invoke_with_args path, each with one query marked skipif evaluate:

Are these changes tested?

cargo test -p datafusion-spark-sqllogictest runs all 257 files in both modes (514 runs). To check which files need the evaluate path, I temporarily ran each failing file record by record, so a file's first failure didn't hide the rest. Every evaluate-mode failure is one of the issues above. The skipif evaluate marks do not change what the PySpark runner runs.

Are there any user-facing changes?

No. CI's sqllogictest job takes longer because every file now runs twice.

@kosiew kosiew left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove, thanks for adding dual-mode coverage. The optimizer approach is appropriate and preserves core-function rewrites.

I think whole-file expected failures might mask unrelated regressions and prevent later records from running. Please scope exclusions to affected records.

known_failure,
) {
(Ok(()), None) => println!("PASS {name}"),
(Err(_), Some(issue)) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whole-file XFAIL accepts unrelated failures and prevents later records from running. Please use issue-linked record-level skips or expectations. Verify that subsequent unbase64 checks execute and mismatches fail the run.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 59ed9f9. EVALUATE_KNOWN_FAILURES is gone. Each record that calls a function with no invoke_with_args yet is marked skipif evaluate, and a note at the top of the file links the issue. The rest of each file now runs in evaluate mode. I checked base64.slt: the unbase64 records run, and changing one expected value fails the run.

- `simplify` uses the default optimizer, so a call whose arguments are all literals,
such as `SELECT expm1(0::INT)`, is constant-folded while planning. This calls the
function's `simplify` (if it has one) and evaluates it on a one-row batch.
- `evaluate` removes the `simplify_expressions` optimizer rule, so every call runs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evaluate mode does not bypass earlier SQL planning, which can simplify table-function arguments. Please narrow the documentation claims and clarify that runtime projections are needed to verify direct evaluation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Narrowed in 59ed9f9. The README now says that evaluate mode only changes the optimizer, that SQL planning can still evaluate some expressions first, such as table function arguments, and that a call over rows is the reliable way to test invoke_with_args.

}
}
println!("{} of {total} test files passed", total - failed);
println!("{} of {total} test runs passed", total - failed);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If XFAIL handling remains, please report expected failures separately from passes. The current summary counts XFAIL runs as passed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the XFAIL handling removed in 59ed9f9, every run is a plain pass or fail.

# A constant scale that is not a bare literal
# The scale is a constant expression, which is a literal only after constant folding,
# so the decimal result type is computed from it only when folding runs.
skipif evaluate

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please link the decimal bround skips to a tracking issue if evaluate mode should support constant-expression scales. Otherwise, document this planner-only limitation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented in 59ed9f9 rather than tracked. The planner computes the decimal result type from literal arguments only, so a scale like 1::bigint is not seen without constant folding. invoke_with_args is fine. A caller that builds the physical expression, such as Comet, passes the result type itself. The comment on each skipped record now says this.

Run every .slt file twice: in simplify mode (default optimizer) and in
evaluate mode (simplify_expressions replaced by a rule that only rewrites
DataFusion's rewrite-only functions), so invoke_with_args is tested for
literal-only calls. Records that cannot run in evaluate mode yet are marked
skipif evaluate with a link to the tracking issue.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove, thanks for addressing the failure-masking issue. Record-level exclusions now preserve coverage without hiding unrelated failures.

The original blocker is resolved, and I see no remaining blocking concerns.

kosiew

This comment was marked as duplicate.

@andygrove
andygrove merged commit ea26e8c into apache:main Nov 19, 2026
10 checks passed
mbutrovich pushed a commit that referenced this pull request Nov 19, 2026
## Which issue does this PR close?

Part of #376.

## Rationale for this change

`main` fails CI since #301 and #316 landed together.
`conditional/nanvl.slt` fails in the new evaluate mode, and the PySpark
runner reports a difference at line 53.

## What changes are included in this PR?

- The seven records that need `nanvl`'s `simplify` rewrite (a lazy expr2
under ANSI mode, and string arguments) are marked `skipif evaluate`,
linking #376.
- Line 53 used `-0.0::double`, which Spark types as a DECIMAL literal
with no negative zero, so Spark returned `false`. It now uses
`'-0.0'::double`, as `log1p.slt` does.

## Are these changes tested?

`nanvl.slt` passes in both modes, and the PySpark runner reports no
differences for it.

## Are there any user-facing changes?

No.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants