Repository navigation
test: run sqllogictests with and without constant folding - #316
Conversation
deeda77 to
7911df6
Compare
There was a problem hiding this comment.
@andygrove, thanks for adding dual-mode coverage. The optimizer approach is appropriate and preserves core-function rewrites.
I think whole-file expected failures might mask unrelated regressions and prevent later records from running. Please scope exclusions to affected records.
| known_failure, | ||
| ) { | ||
| (Ok(()), None) => println!("PASS {name}"), | ||
| (Err(_), Some(issue)) => { |
There was a problem hiding this comment.
Whole-file XFAIL accepts unrelated failures and prevents later records from running. Please use issue-linked record-level skips or expectations. Verify that subsequent unbase64 checks execute and mismatches fail the run.
There was a problem hiding this comment.
Done in 59ed9f9. EVALUATE_KNOWN_FAILURES is gone. Each record that calls a function with no invoke_with_args yet is marked skipif evaluate, and a note at the top of the file links the issue. The rest of each file now runs in evaluate mode. I checked base64.slt: the unbase64 records run, and changing one expected value fails the run.
| - `simplify` uses the default optimizer, so a call whose arguments are all literals, | ||
| such as `SELECT expm1(0::INT)`, is constant-folded while planning. This calls the | ||
| function's `simplify` (if it has one) and evaluates it on a one-row batch. | ||
| - `evaluate` removes the `simplify_expressions` optimizer rule, so every call runs |
There was a problem hiding this comment.
Evaluate mode does not bypass earlier SQL planning, which can simplify table-function arguments. Please narrow the documentation claims and clarify that runtime projections are needed to verify direct evaluation.
There was a problem hiding this comment.
Narrowed in 59ed9f9. The README now says that evaluate mode only changes the optimizer, that SQL planning can still evaluate some expressions first, such as table function arguments, and that a call over rows is the reliable way to test invoke_with_args.
| } | ||
| } | ||
| println!("{} of {total} test files passed", total - failed); | ||
| println!("{} of {total} test runs passed", total - failed); |
There was a problem hiding this comment.
If XFAIL handling remains, please report expected failures separately from passes. The current summary counts XFAIL runs as passed.
There was a problem hiding this comment.
With the XFAIL handling removed in 59ed9f9, every run is a plain pass or fail.
| # A constant scale that is not a bare literal | ||
| # The scale is a constant expression, which is a literal only after constant folding, | ||
| # so the decimal result type is computed from it only when folding runs. | ||
| skipif evaluate |
There was a problem hiding this comment.
Please link the decimal bround skips to a tracking issue if evaluate mode should support constant-expression scales. Otherwise, document this planner-only limitation.
There was a problem hiding this comment.
Documented in 59ed9f9 rather than tracked. The planner computes the decimal result type from literal arguments only, so a scale like 1::bigint is not seen without constant folding. invoke_with_args is fine. A caller that builds the physical expression, such as Comet, passes the result type itself. The comment on each skipped record now says this.
Run every .slt file twice: in simplify mode (default optimizer) and in evaluate mode (simplify_expressions replaced by a rule that only rewrites DataFusion's rewrite-only functions), so invoke_with_args is tested for literal-only calls. Records that cannot run in evaluate mode yet are marked skipif evaluate with a link to the tracking issue. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
59ed9f9 to
043ca1f
Compare
kosiew
left a comment
There was a problem hiding this comment.
@andygrove, thanks for addressing the failure-masking issue. Record-level exclusions now preserve coverage without hiding unrelated failures.
The original blocker is resolved, and I see no remaining blocking concerns.
## Which issue does this PR close? Part of #376. ## Rationale for this change `main` fails CI since #301 and #316 landed together. `conditional/nanvl.slt` fails in the new evaluate mode, and the PySpark runner reports a difference at line 53. ## What changes are included in this PR? - The seven records that need `nanvl`'s `simplify` rewrite (a lazy expr2 under ANSI mode, and string arguments) are marked `skipif evaluate`, linking #376. - Line 53 used `-0.0::double`, which Spark types as a DECIMAL literal with no negative zero, so Spark returned `false`. It now uses `'-0.0'::double`, as `log1p.slt` does. ## Are these changes tested? `nanvl.slt` passes in both modes, and the PySpark runner reports no differences for it. ## Are there any user-facing changes? No.
Which issue does this PR close?
None. It adds the test coverage that found #305–#315.
Rationale for this change
A Spark function can run in two ways. When every argument is a literal, the
simplify_expressionsoptimizer rule constant-folds the call while planning. That callsScalarUDFImpl::simplify(if the function has one) and evaluates the result on a one-row batch. Otherwise the call runs throughinvoke_with_argsduring execution. That is also the only path for engines such as Comet that build physical expressions directly.Most
.sltqueries are literal-only, such asSELECT f(1), so today they only test the first path.invoke_with_argsmust always work, andsimplifyis an optional optimization. This PR runs the tests through both paths.What changes are included in this PR?
.sltfile twice:simplify: the default optimizer, as before.evaluate: the same optimizer rules withsimplify_expressionsreplaced by a rule that rewrites only DataFusion's rewrite-only functions (arrow_cast,arrow_try_cast,cast_to_type,try_cast_to_type,nvl2). The tests use those to build typed inputs, and theirinvoke_with_argsalways errors. Every other call, including literal-only calls of the Spark functions, runs throughinvoke_with_args..sltlabel, so a record can be markedskipif evaluate. (skipifrather thanonlyif simplify, because the PySpark runner skipsonlyifrecords that are not forspark.)--mode=simplifyor--mode=evaluateruns a single mode. Results print asPASS [evaluate] path.evaluatemode yet is markedskipif evaluate, with a note at the top of the file linking its issue.--completeruns only insimplifymode, so each file is rewritten once.test_files/README.mddocuments the two modes.What
evaluatemode foundEight functions have no working
invoke_with_args: they error unlesssimplifyrewrites them first. Their records are markedskipif evaluate.ifunix_datedate_diffdate_trunctime_trunctruncadd_monthsbase64Two bugs in the
invoke_with_argspath, each with one query markedskipif evaluate:concat_wspanics when all arguments are scalars andnumber_rows> 1 #314:concat_wspanics when all arguments are scalars andnumber_rows> 1. This is the same pattern feat(spark): add Spark-compatible hash (Murmur3) function #68 fixed inhash.make_dt_intervaldeclares a non-nullable result but can return NULL #315:make_dt_intervaldeclares a non-nullable result but can return NULL.Are these changes tested?
cargo test -p datafusion-spark-sqllogictestruns all 257 files in both modes (514 runs). To check which files need the evaluate path, I temporarily ran each failing file record by record, so a file's first failure didn't hide the rest. Every evaluate-mode failure is one of the issues above. Theskipif evaluatemarks do not change what the PySpark runner runs.Are there any user-facing changes?
No. CI's sqllogictest job takes longer because every file now runs twice.