Skip to content

feat(spark): cast sha1's argument to Spark's declared input type - #355

Merged
andygrove merged 1 commit into
apache:mainfrom
andygrove:sha1-input-types
Nov 18, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
andygrove:sha1-input-types

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

Spark declares sha1's input type as BINARY, and casts a STRING argument to it. sha1 already accepts the same types, but it rejects the others with DataFusion's errors instead of Spark's:

Query Before This PR
sha1(1::INT) Function 'sha1' requires Binary, but received Int32 ... [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `sha1` requires the "BINARY" type, however it has the type "INT". SQLSTATE: 42K09
sha1('a', 'b') Function 'sha1' expects 1 arguments but received 2 ... [WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `sha1` requires 1 parameters but the actual number is 2. ...

What changes are included in this PR?

sha1 declares Spark's Sha1.inputTypes with SparkInputTypes from #350, the same way sha2 does.

Are these changes tested?

Yes. New records in hash/sha1.slt cover a dictionary-encoded string, a column with NULLs, the errors for other types and for the wrong number of arguments, and ANSI mode. The error records fail without this change. I checked sha1.slt against Spark 4.2.0 with the PySpark runner, and every record matches. The runner does not compare error messages, so I checked them against Spark by hand. Spark's message also names the argument's SQL text, as noted in #350.

Are there any user-facing changes?

Only the error messages above. The results do not change.

This PR was prepared with AI assistance (Claude Code) and reviewed by me.

Spark declares sha1's input type as BINARY. Use SparkInputTypes so a
STRING argument is cast to BINARY and other types fail with Spark's
UNEXPECTED_INPUT_TYPE and WRONG_NUM_ARGS errors.

Part of apache#353.

@coderfender coderfender left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andygrove
andygrove merged commit 835af64 into apache:main Nov 18, 2026
8 checks passed
@andygrove
andygrove deleted the sha1-input-types branch November 18, 2026 22:02
@andygrove

Copy link
Copy Markdown
Member Author

Thanks @coderfender!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants