Repository navigation
fix(query_engine): treat Unicode whitespace as whitespace and report non-ASCII errors - #1816
Merged
fengelniederhammer merged 5 commits intoNov 18, 2026
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the Unicode lexer changes.
No open issues 🎉
What changed in this PR
Updates the SaneQL lexer to handle Unicode whitespace and produce readable errors for unsupported characters and invalid UTF-8, addressing #1791.
Changes:
- Accepts Unicode whitespace and ignores a leading UTF-8 byte order mark.
- Validates non-ASCII input, reports ASCII-safe errors, and counts columns by code point.
- Adds lexer and query-level regression tests and documents UTF-8 requirements.
| File | Description |
|---|---|
src/rhydb/query_engine/saneql/lexer.test.cpp |
Tests Unicode whitespace, UTF-8 validation, errors, and column tracking. |
src/rhydb/query_engine/saneql/lexer.h |
Declares code-point decoding and whitespace helpers. |
src/rhydb/query_engine/saneql/lexer.cpp |
Implements Unicode handling, BOM skipping, and clear diagnostics. |
src/rhydb/query_engine/saneql/lexer_unicode.test.cpp |
Adds query-level tests for Unicode input and invalid UTF-8. |
documentation/query_documentation.md |
Documents UTF-8, leading BOM, and Unicode whitespace support. |
Review effort: Balanced
fengelniederhammer
marked this pull request as ready for review
November 11, 2026 08:17
Contributor
|
This is a preview of the changelog of the next release. If this branch is not up-to-date with the current main branch, the changelog may not be accurate. Rebase your branch on the main branch to get the most accurate changelog. Note that this might contain changes that are on main, but not yet released. Changelog: 0.17.6 (2026-11-18)FeaturesBug Fixes |
Collaborator
|
Do we know why we did not get a proper error message before? The result was not even valid json |
taepper
reviewed
Nov 16, 2026
…characters-outside-string-literals
…characters-outside-string-literals
fengelniederhammer
deleted the
1791-queries-can-fail-with-non-ascii-characters-outside-string-literals
branch
November 18, 2026 09:13
This was referenced Nov 18, 2026
taepper
pushed a commit
that referenced
this pull request
Nov 18, 2026
🤖 I have created a release *beep* *boop* --- ## [0.17.6](v0.17.5...v0.17.6) (2026-11-18) ### Features * **query_engine:** add type casts at query execution time ([#1799](#1799)) ([f7acdf0](f7acdf0)) ### Bug Fixes * **query_engine:** do not decompress every row before applying a limit ([#1833](#1833)) ([d3c06a2](d3c06a2)) * **query_engine:** evaluate out-of-bound `.at()` expressions to null for null inputs ([#1839](#1839)) ([511d2ee](511d2ee)) * **query_engine:** limit the nesting of queries ([#1828](#1828)) ([599d3a4](599d3a4)) * **query_engine:** treat Unicode whitespace as whitespace and report non-ASCII errors ([#1816](#1816)) ([edc1c02](edc1c02)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
resolves #1791
Summary
Queries with a non-breaking space (or other Unicode whitespace) outside string literals no longer fail to parse, and non-ASCII / invalid-UTF-8 input now produces clear error messages instead of a garbled 400 response body.
White_Spaceset (nbsp, em space, ideographic space, NEL, etc.), not just ASCII space/tab/newline. A leading UTF-8 BOM is ignored.Unexpected character U+XXXX; invalid UTF-8 withInvalid UTF-8 byte 0xNN. Both are plain ASCII, so the 400 body stays valid UTF-8. Invalid UTF-8 is rejected everywhere, including inside string literals and comments.PR Checklist