Skip to content

fix(spark): url_decode replaces invalid utf-8 like spark - #372

Open
TechWizard9999 wants to merge 3 commits into
apache:mainfrom
TechWizard9999:fix/url-decode-invalid-utf8
Open

TechWizard9999 wants to merge 3 commits into
apache:mainfrom
TechWizard9999:fix/url-decode-invalid-utf8

Conversation

@TechWizard9999

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Explained in the issue.

What changes are included in this PR?

Bytes that are not valid utf-8 become the replacement character, like spark. a bad percent escape is still an error.

Are these changes tested?

Yes, updated the slt tests for both functions.

Are there any user-facing changes?

Yes, invalid utf-8 gives the replacement character instead of an error or null.

@TechWizard9999

Copy link
Copy Markdown
Contributor Author

@andygrove Please review. Thanks.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The fix is in the right place and the #[user_doc] update and the removed xfail markers are just what #366 needed. I rebased onto main locally and all the url/ files pass in both modes, and the PySpark runner agrees on every record. I left one comment about encoded surrogates, where Java's decoder still gives a different number of U+FFFD.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] url_decode raises an error for invalid UTF-8, Spark replaces it with U+FFFD

2 participants