Repository navigation
Conversation
Register find_in_set with Spark string coercion and an INT result for every string layout. Reuse the core search kernel and add SQL coverage for nulls, casts, mixed string layouts, and the return type.
andygrove
left a comment
There was a problem hiding this comment.
Thanks for this. Reusing the core kernel and only fixing the layout and return type keeps it small, and the mixed-layout and arrow_typeof tests pin exactly the parts that differ from core. I rebased it onto main locally (to pick up #316) and the file passes in both the simplify and evaluate modes. The PySpark runner agrees on every record except the marked collation one.
| Int32 | ||
| Int32 | ||
|
|
||
| # xfail spark: DataFusion SQL does not support explicit collations. |
There was a problem hiding this comment.
Could this marker link the collation tracking issue, so it gets cleaned up when collations land? Something like # xfail spark: collations are not supported yet (https://github.com/apache/datafusion-spark/issues/299).
|
|
||
| # Spark implicitly casts both arguments to strings. | ||
| query II | ||
| SELECT find_in_set(2::INT, '1,2,3'), find_in_set('2', 2::INT); |
There was a problem hiding this comment.
Nice coverage of the int and boolean casts. Binary and date go through different branches of implicit_cast, so it might be worth pinning one of each too. find_in_set(X'62', 'a,b') and find_in_set(DATE '2020-01-01', 'x,2020-01-01') both return 2 here and on Spark 4.2.0.
|
thanks, i linked #299 in the collation comment and added the binary and date cases. the find_in_set tests pass in both simplify and evaluate modes. |
Which issue does this PR close?
Closes #200.
What changes are included in this PR?
Register
find_in_setwith Spark’s string coercion rules and reuse DataFusion’s existing search implementation. The result is alwaysINT, including forLargeUtf8inputs.Are these changes tested?
Added SQL cases for nulls, implicit casts, mixed string layouts, and the return type. All 562 Rust tests and 258 SQL test files passed. Formatting and Clippy passed. I also checked 20 cases against Spark 4.2.0’s string implementation.
Are there any user-facing changes?
find_in_setnow uses Spark-compatible behavior for the default binary collation.