regex_match_dyn (array-array path) does not support Dictionary-encoded needle arrays (SIMILAR TO / ~ operators)

[u70b3]

Describe the bug

The regex kernels' array-array path (regex_match_dyn) does not support Dictionary(_, Utf8)-encoded needle arrays, so SIMILAR TO (and the ~ regex operators) fail with an internal error when the value side is dictionary-encoded and the pattern is a non-scalar array:

CREATE TABLE t AS SELECT * FROM (VALUES ('user auth failed')) v(s);
CREATE TABLE p AS SELECT * FROM (VALUES ('(auth|login)')) v(pat);
SELECT arrow_cast(t.s, 'Dictionary(Int32, Utf8)') SIMILAR TO p.pat FROM t CROSS JOIN p;

Internal error: Data type Dictionary(Int32, Utf8) not supported for regex_match_dyn on string array.

Expected behavior

The query should evaluate successfully (return true for this data), like the equivalent LIKE query already does:

SELECT arrow_cast(t.s, 'Dictionary(Int32, Utf8)') LIKE p.pat FROM t CROSS JOIN p;
-- returns true

Additional context

• The scalar fast path (regex_match_dyn_scalar) already handles Dictionary arrays (see datafusion/physical-expr/src/expressions/binary/kernels.rs), and the LIKE kernels handle dictionary needles in the array-array path, so this is a gap specific to the regex kernels' array-array dispatch.
• Suggested fix: add a DataType::Dictionary arm to regex_match_dyn that matches over the dictionary value type (mirroring the scalar path / like kernels).
• Surfaced during review of fix: coerce SIMILAR TO operands to a common string type #23704 (which fixed the missing type coercion for SIMILAR TO and turned the former panic paths into this proper error). Not a regression from that PR — the limitation is pre-existing.

[Jithendra2608]

take

[u70b3]

Hi @Jithendra2608 — that was quick, thanks for picking this up! I had originally planned to implement this one myself as a follow-up to #23704 (where the idea came up during review), but I'm glad to see it in good hands — it's all yours. If any context from that discussion would be useful, feel free to ping me. Good luck, looking forward to the PR!

[Jithendra2608]

Hi @u70b3 , thank you for the warm welcome and the excellent context in the issue description!
I actually just finished putting together the implementation and opened the PR here: #23722. I mirrored the dictionary unpacking logic from the scalar path exactly as you suggested.
Since you have the context from #23704, I would absolutely love your feedback or review on the PR whenever you have a free moment. Thanks again!

[adriangb]

@u70b3 review on other PRs is very welcome - it's our biggest bottleneck and a great way to get involved in the community

[u70b3]

Status update from reviewing #23722: this repro is already fixed at the SQL level on current main — as a side effect of #23704, which added regex_coercion for Expr::SimilarTo in the analyzer. The dictionary value side now gets a decode cast before planning reaches the regex kernel:

EXPLAIN SELECT arrow_cast(t.s, 'Dictionary(Int32, Utf8)') SIMILAR TO p.pat FROM t CROSS JOIN p;
-- ProjectionExec: expr=[CAST(CAST(s@0 AS Dictionary(Int32, Utf8)) AS Utf8) ~ pat@1 ...]

I verified the original repro above returns true on main (c111f0a) without any further patch. The same holds for ~/~*/!~* and the regexp UDFs, which were already coerced via casts.

What remains is kernel-level robustness: physical plans that bypass the analyzer (hand-built BinaryExpr, FFI/Substrait consumers) can still deliver a Dictionary array to regex_match_dyn and hit the internal_err! reported here. That is the scope #23722 now covers — details in my review there. So this issue can be closed either way: as fixed by #23704, or once #23722 lands with the arm covered by a direct unit test.

[Jithendra2608]

Hi @u70b3, thank you for the incredible review and for providing the unit test snippet!
I have updated the PR with your feedback:
1.Restored the removed doc-comments (which also fixed the cargo fmt issue).
2.Cleaned up the take import.
3.Added the direct unit test for the Dictionary arm.
4.Rebased the branch onto the latest main.